Force feedback provided in web pages

ABSTRACT

Force feedback is provided to a user of a client computer receiving information such as a web page over a network such as the World Wide Web from a server machine. The client machine has a force feedback interface device through which the user experiences physical force feedback. The web page may include force feedback information to provide authored force effects. Force feedback is correlated to web page objects by a force feedback program running on the client and based on input information from the interface device, the web page objects, and the force feedback information. Generic force effects can also be provided, which are applied uniformly at the client machine to all web page objects of a particular type as defined by user preferences at the client machine. A web page authoring interface is also described that includes the ability to add force sensations to a web page. The user may assign force effects to web page objects and immediately feel how the web page will feel to an end user. A web page is output by the interface, including force information to provide the force effects at a client. The authoring tool can include or access a force design interface for creating or modifying force effects.

CROSS REFERENCE TO RELATED APPLICATIONS

This is a continuation application of prior U.S. application Ser. No.09/244,622, now U.S. Pat. No. 6,161,126, filed on Feb. 2, 1999, which 1)is a continuation-in-part of U.S. application Ser. No. 08/691,852, nowU.S. Pat. No. 5,956,484, filed Aug. 1, 1996, which is acontinuation-in-part of U.S. patent application Ser. No. 08/571,606, nowU.S. Pat. No. 6,219,032, filed Dec. 13, 1995; 2) is acontinuation-in-part of application Ser. No. 08/970,953, filed Nov. 14,1997; and 3) claims priority to U.S. Provisional patent application No.60/073,518, filed Feb. 3, 1998; the disclosures of which are allincorporated herein by reference.

This application contains a Microfiche Appendix which contains 1Microfiche having 1 sheet, 36 Frames.

BACKGROUND OF THE INVENTION

This invention relates generally to human/computer interfaces, and moreparticularly to human/computer interfaces with force feedback that canoperate over a network.

The Internet has, of late, become extremely popular. While the use ofthe Internet has been prevalent for many years now, its use has beenlimited by the arcane and difficult commands required to access thevarious computers on the network. To address this problem, a protocolknown as the “World Wide Web” or “WWW” was developed to provide aneasier and user-friendlier interface for the Internet. With the WorldWidE Web, an entity having a domain name creates a “web page” or “page”which can provide information and, to a limited degree, someinteractivity.

A computer user can “browse”, i.e. navigate around, the WWW by utilizinga suitable web browser and a network gateway (e.g., an Internet ServiceProvider (ISP)). Currently, popular web browsers, include Netscape®Navigator® made by Netscape Corporation of Mountain View, Calif., andInternet Explorer made by Microsoft® Corporation of Redmond, Wash. A webbrowser allows a user to specify or search for a web page on the WWW,and then retrieves and displays web pages on the user's computer screen.

The Internet is based upon a transmission protocol known as“Transmission Control Protocol/Internet Protocol” (or “TCP/IP” forshort), which sends “packets” of data between a host machine, e.g. aserver computer on the Internet, and a client machine, e.g. a user'spersonal computer connected to the Internet. The WWW is an Internetinterface protocol which is supported by the same TCP/IP transmissionprotocol. Intranets are private networks based upon Internet standards;since they adhere to Internet standards, can often use the same webbrowser software and web server software as are used on the Internet.

A web page typically includes static images, animated images (e.g.video), and/or text. The images and text are specified in a “HyperTextMark-up Language” (“HTML”) file that is sent from the web server to theclient machine. This HTML file is parsed by the web browser in order todisplay the text and images on the display of the client machine. Otherstandardized languages or protocols are also being developed for usewith the Internet and the World Wide Web. For example, the VirtualReality Modeling Language (VRML) is used to provide visual virtual 3-Denvironments and allow one or many users to navigate through andinteract as “avatars” in such an environment using a web browser orother software on a client computer system.

Furthermore, additional functionality may be provided in web pages withinformation downloaded over the WWW in the form of scripts or programs.Scripting, typically in the form of VBScript or JavaScript, allows aseries of instructions to be perfonned on the client computer once theinstructions have been downloaded in a web page. Programs can beprovided in such standard languages as Visual Basic, C++, or currentlyin the form of Java “applets” or ActiveX® controls. Java includes aplatform-independent interpreter running on the client machine thatexecutes downloaded applets within web pages or other programs, e.g., todisplay animated images, retrieve data over the WWW, output feedback tothe user, or perform operating system tasks. ActiveX controls similarlyexecute on the client computer once the program instructions areresident on the client. ActiveX controls are programmable objects thatcan be embedded into Web pages and may be written in any(platform-specific) language. Java and ActiveX controls can addfunctionality to a Web page that would normally be difficult, or evenimpossible, using HTML or scripting languages. ActiveX controls can alsobe controlled with a scripting language. Alternatively, functionalitycan be added to a web page through the use of “plug-ins”, which areapplication programs running in conjunction with certain web browsers toparse plug-in-specific code in the web page which the browser cannotunderstand.

Other WWW-related functionality includes Dynamic HTML. Dynamic HTML is aset of features currently incorporated in browsers such as MicrosoftInternet Explorer that enable authors to dynamically change therendering and content of an HTML document. Using Dynamic HTML, a contentdeveloper or programmer can access the attributes of a document'scontents or objects (such as an object's position on the page and type).In addition, event messages are generated when a user interacts with theweb page content (such as when a user clicks on a graphical buttonimage). The features of Dynamic HTML can be elicited through the use ofVBScript or JavaScript scripts embedded in a Web page orprogrammatically through Visual Basic or C++.

The Internet and the WWW also permit sound data to be transmitted overthe Internet. For example, references to sound files can be embedded inHTML pages and can be played by the web browser. Data “packets” coded inTCP/IP format can also be sent from one client machine to another overthe Internet to transmit sound data. This last-mentioned technique formsthe basis for Internet telephony.

While the transmission of visual images (both static and dynamic), text,and sound over the Internet is well-known, the transmission of othertypes of sensory data has not been well explored. In particular, thetransmission of data over the Internet pertaining to the sense of touchand/or force has not been established. “Force feedback” allows a user toexperience or “feel” tactile sensations as provided throughcomputational information. Using computer-controlled actuators andsensors on a force feedback device, a variety of realistic sensationscan be modeled and experienced by the user. This useful and highlyimmersive sensory modality for interacting with the Internet has heretobeen unavailable.

In addition, there needs to be tools to allow web page authors toquickly and easily add forces to web page content or adjust/modifyexisting forces as desired. Such tools should preferably allow an authoror user to intuitively include forces in web pages without needing aknowledge of force feedback instructions or programming constructs. Inaddition, the author should be provided with an intuitive way to designor adjust the forces. Thus, a tool is needed for assisting the user,programmer or developer in intuitively and easily creating web pages andin setting force feedback characteristics to provide desired forcesensations in web pages.

SUMMARY OF THE INVENTION

The present invention is related to the transmission/reception ofinformation pertaining to force feedback to provide feel in transmittedinformation such as web pages. The force feedback provided by themethods and apparatus of the present invention enhance the sensoryexperience of the user to provide a richer, more interesting, and moreenjoyable interaction with information received over networks.

In one aspect of the present invention for providing force feedback overa network, information is received by a client machine over a networkfrom a server machine. The client machine (typically a personalcomputer) has a visual display and a force feedback interface device.Force feedback interface devices can include force feedback mice,joysticks, finger cradles, trackballs, steering wheels, and yokes. Theforce feedback interface device includes sensors and actuators andpreferably a local microprocessor for communicating with the clientmachine and controlling the actuators of the interface device.

Several embodiments are disclosed in which forces are implemented basedon the information received over the network. The disclosed embodimentsemphasize the World Wide Web or the Internet as the network providedbetween client and server computers, where web page information receivedby the client is in HTML or other standard format and a web browserrunning on the client machine displays the web page. One embodimentprovides authored force effects, which are effects particular to a webpage that are customized for each web page object and are downloadedwith the web page. The received web page information includes screendisplay information representing a visual layout of a web page and forcefeedback information related to providing a feel sensation. Inputinformation from the interface device is used to position a cursor withrespect to the visual layout of the web page. A force feedback signal isprovided based upon the input information and the web page information,and the received force feedback information includes a call to a forcefeedback program running on the client machine that provides the forcefeedback signal. The force feedback signal is received by the interfacedevice, which outputs computer-controlled physical force feedback to theuser correlated with the visual layout of the web page. The forcefeedback program running on the client can be ActiveX control, a Javaapplet, a plug-in, etc. In one preferred embodiment, the ActiveX controlis called by script instructions, e.g. in JavaScript, included in thereceived force feedback information. The script instructions can providedownloaded force effect parameters to the force feedback program tooutput a desired force effect.

In a different embodiment, generic force effects are implemented.Generic effects are applied uniformly to all web page objects of aparticular type. A web page need not include force information to allowgeneric force effects to be applied to the web page content. Thus, areceived web page includes screen display information representing webpage objects. The client determines which web page objects are force webpage objects to be associated with at least one generic force effect,where the force web page objects are of a predefined type. A genericforce effect is assigned to each type of web page object as defined byeffect information derived from the client machine. Generic forceeffects are output when a user-controlled cursor interacts with a forceweb page object. A force control program running on the client machinecan detect whether the cursor is contacting one of the force web pageobjects and output the appropriate force feedback signals to theinterface device. In another embodiment, a downloaded web page can bepreprocessed before being displayed to add force feedback information tothe web page. The client can perform the preprocessing, or a proxyserver can preprocess the web page before sending the web page to theclient. In other embodiments, both generic effects and authored effectscan be provided for a particular web page. If both types of forceeffects are used, either type of force effect can be overridden by theother depending on user preferences.

In another aspect of the present invention, a web page authoringinterface is provided that includes the ability to add force sensationsto a web page. The web page authoring interface is displayed on adisplay device of a computer and displays a web page including web pageobjects. Input from a user is received to the authoring interface andselects a web page object and a force effect to be associated with theselected web page object. A web page is output by the interface,including the web page objects and including force information to allowthe force effect to be implemented when the web page is displayed by aclient machine after being received from a server machine over anetwork. The authoring tool can include a force design interface forcreating or modifying the force effect, or a separate force designinterface can accessed. The force effects added to web page objects canalso be experienced directly by the author during the design process. Inone embodiment, the author is also able to spatially designate an areaof a web page object to be associated with a selected force effect, e.g.a graphical mark or outline can designate sub-areas of an image or textto be associated with forces. In addition, the user can preferablyassociate sound data with a web page object, such that when said forceeffect is output, a sound is output to the user synchronized with theforce effect. Finally, a predefined graphical identifier can be insertedinto the web page to indicate to a user that the web page provides forceeffects, and which may also be a link that causes a linked forcefeedback resource web page to be downloaded.

The present invention provides several ways to implement force feedbackover a network such as the World Wide Web. Embodiments included hereinallow standard web pages without any force content to be assigned forceson a client machine including force feedback capability. In otherembodiments, a web page author can include specific force effects in aweb page to any desired level of customization. The described forcefeedback web page authoring tool allows force effects to be easily andquickly included in web pages to foster greater diversity and widespreaduse of feel in web pages.

These and other advantages of the present invention will become apparentupon reading the following detailed descriptions and studying thevarious figures of the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a pictorial representation of the Internet, a web servermachine, and two client machines;

FIG. 2 is a block-diagram of a client machine used in the presentinvention;

FIG. 3 is a block-diagram of a force feedback system in accordance withthe present invention;

FIG. 4a is a perspective view of a preferred human/computer interface(“force feedback device”) of the present invention;

FIG. 4b is a perspective view of the mechanism of the force feedbackdevice of FIG. 4a;

FIG. 5 is a block diagram of a wide area network (WAN) based uponInternet TCP/IP protocol and supporting World Wide Web (WWW) HTMLprotocols in accordance with the present invention;

FIG. 6 is an illustration of a web browser and an image displayed on avisual display of a client computer as generated from a downloaded HTMLweb page file;

FIG. 7 is a flow-diagram illustrating a first embodiment of anembodiment using Dynamic HTML to provide force effects in web pages;

FIG. 8 is a diagrammatic illustration of the coordinate frames providedin Dynamic HTML;

FIG. 9 is a diagrammatic illustration of coordinates used in atransformation to obtain screen coordinates of an object;

FIG. 10 is a flow-diagram of illustrating a second embodiment of anembodiment using Dynamic HTML to provide force effects in web pages;

FIGS. 11a and 11 b are diagrammatic illustrations of the two embodimentsshown in FIGS. 7 and 10;

FIGS. 12, 13 a, 13 b, 14, and 15 are illustrations of displayed webpages in which web page objects are associated with authored forceeffects using an ActiveX control;

FIG. 16 is a diagrammatic illustration showing a proxy server forproviding preprocessing to web pages to add force effects;

FIG. 17a and 17 b are illustrations of embodiments of web page authoringapplications of the present invention that includes force feedbackediting functionality;

FIG. 18 is a block diagram illustrating the web page authoring interfaceof the present invention and web pages produced by the authoringinterface;

FIGS. 19a, 19 b, and 19 c are illustrations showing force effect editinginterface examples which can be used in conjunction the authoringapplication of FIGS. 17a-17 b;

FIGS. 20-22 illustrate a web page editor displaying an image in a webpage and providing an outline function to add force effects to portionsof the image;

FIGS. 23 and 24 illustrate the web page editor of FIGS. 20-22 addingforce effects to a different image;

FIG. 25 is a diagrammatic illustration showing features of a firstembodiment of the force feedback web page editor of the presentinvention; and

FIG. 26 is a diagrammatic illustration showing features of a secondembodiment of the force feedback web page editor of the presentinvention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

In FIG. 1, a network system 10 includes a wide area network (WAN) suchas the Internet 12, and a number of computers or “machines” coupled tothe Internet 12. For example, a first client machine 14, a second clientmachine 16, and a web server machine 18, are coupled to the Internet 12.

As noted previously, both the Internet 12 and Intranets operate usingthe same TCP/IP protocols. This allows Intranets to use similar or thesame server machine software and client machine software as are used inInternet 12 applications. The Internet 12 includes a number of nodes 20that are interconnected by data transmission media 22. These nodes aretypically routers, switches, and other intelligent data transmissionapparatus which route “packets” of TCP/IP information to the desireddestination. In some instances, the nodes 20 comprise an Internetservice provider (ISP) 20a which allows a client machine to access the“backbone” of the Internet. Alternatively, client machines and webservers can be coupled directly into the backbone of the Internet.

As noted previously, the present invention is directed to theimplementation of force feedback over a network, such as the Internet12. To provide a user of a client machine with the experience of forcefeedback, force feedback human/computer interfaces (hereafter “forcefeedback devices”) 24 and 26 can be provided as part of the clientmachines 14 and 16, respectively. The client machines 14 and 16 aretypically provided with computer video monitors 28 and 30 (which is oneexample of a “visual display”), respectively, which can display imagesI1 and I2, respectively. Preferably, forces developed by force feedbackdevices 24 and 26 are correlated with the images I1 and I2 of the clientmachines 14 and 16, respectively.

The machines 14-18 are considered, in the language of the Internet, tobe “resources,” and each has its own unique Uniform Resource Locator or“URL.” In one embodiment of the present invention, a client machine,such as client machine 14 or 16, sends a request for a “web page”residing on, for example, web server machine 18. This is accomplished bythe client machine sending a connection request and a URL whichspecifies the address of the web page to the web server machine 18. Theweb server machine 18 then sends a web page 32 in HTML format back tothe requesting client machine where it is “cached” in the memory(typically the RAM, hard disk, or a combination of the two) of theclient machine. In this embodiment of the invention, the image on thevideo display of the client machine is generated from the HTML web pagefile cached on the client machine, and force feedback is provided to auser through the force feedback device as he manipulates a usermanipulable object of the force feedback device. The embodiments of thepresent invention can also be used with other collections of data ordocuments besides web pages.

In another aspect of the present invention, a first client machine, suchas client machine 14, and a second client machine, such as clientmachine 16, directly communicate force feedback commands to each otherin standard TCP/IP protocol over the Internet 12. More particularly,client machine 14 can send force feedback and other information to theURL of the client machine 16, and the client machine 16 can send forcefeedback and other information in standard TCP/IP packets to the URL ofthe client machine 14. In this way, users of client machine 14 andclient machine 16 can interact physically over the Internet 12. Ofcourse, a server machine 18 can likewise directly communicate forcefeedback commands to a client machine 12 or 14, or all three machinescan interact.

In FIG. 2, a “personal” computer 34 architecture that can be used forclient machine 14 or client machine 16 is shown in block diagram form.It should be noted that a variety of machine architectures can be usedto access the Internet 12, such as a PC compatible computer under theWindows or MS-DOS operating system, Macintosh personal computer, SUN orSilicon Graphics workstation, home video game systems (such as systemsavailable from Nintendo, Sega, or Sony), “set top box”, “network accesscomputers”, portable/handheld computers of all types, or anygeneral-purpose computer. The particular architecture shown for thecomputer 34 is a typical personal or “PC” computer architecture. Webserver machines can also have similar architectures, but are often morepowerful “workstations” that, for example, operate under some variant ofthe UNIX® operating system. The Internet service providers 20 a arelikewise often UNIX-based computers or powerftil personal computersrunning Windows NT®. The nodes 20 are most commonly routers built byCisco Systems of San Jose, Calif. Client machine 14 or 16 can also takeother forms, such as a television including or connected to amicroprocessor for Internet access. Force feedback devices used withsuch client machines can be appropriate for the particular embodiment,e.g., a TV remote control used for internet browsing on theabovementioned television can include force feedback functionality.

The personal computer system 34 includes a microprocessor 36 clocked bya system clock CLK and which is coupled to a high speed or memory bus 38and to a lower speed or I/O bus 40. The system RAM 42 and ROM 44 aretypically coupled to the high speed memory bus, while variousperipherals, such as the video display, hard disk drive, Internetinterface (often either a modem or an Ethernet connection), and forcefeedback device, are typically coupled to the slower I/O bus. Themicroprocessor executes programs stored in the various memories or othercomputer-readable medium of the computer 34 (RAM, ROM, hard disk, signalpropagated by a carrier wave, etc.) to control, for example, the imagedisplay on the video display and the forces provided by the forcefeedback device. The manufacture and use of computers, such as personalcomputer 34, are well-known to those skilled in the art.

In FIG. 3, a client machine 46 in accordance with the present inventionincludes a personal computer system 48 and a force feedbackhuman/computer interface or “force feedback device” 50. A user 52 canreceive visual information 54 and auditory information 56 from thepersonal computer 48 and can manipulate the force feedback device 50 asindicated at 58 a and 58 b to provide input, e.g., to command a cursorlocation on a visual display or other provide other control information.In addition, the user 52 can receive force feedback 60 from the forcefeedback device 50 to represent physical “feel” or force sensations.

The personal computer system 48 includes the microprocessor 36, thesystem clock 62, a video monitor 64 (which is one type of “visualdisplay”), and an audio device 66. The system clock 62, as explainedpreviously, provides a system clock signal CLK to the microprocessor 36and to other components of the personal computer system 48. The displaydevice 64 and the audio output device 66 are typically coupled to theI/O bus 40 (not shown in this figure).

In this preferred embodiment, the force feedback device 50 preferablyincludes a local microprocessor 68, a local clock 70, optional localmemory 71 for the local microprocessor 68, a sensor interface 72,sensors 74, a user manipulatable object 76, “other” input interface 78,an actuator interface 80, a safety switch 82, and actuators 84 whichprovide a force F to the object 76, and an optional power supply 86 toprovide power for the actuator interface 80 and actuator 84.

The microprocessor 36 of the personal computer system 48 is coupled forcommunication with the local microprocessor 68 of the force feedbackdevice 50. This communication coupling can be through a serial portcoupling 88 to the personal computer system. A preferred communicationcoupling the Universal Serial Bus (USB) of a personal computer, althoughan RS-232 serial bus, or other communication coupling (such asFirewire), a parallel bus, an Ethernet bus, or other types of interfacesor communication links can also be used.

In use, the user 52 of the client machine 46 grasps the object 76 of theforce feedback device 50 and manipulates (i.e. exerts a force to move orattempt to move) the object to cause a “pointer” icon (cursor) to movein the image displayed by the display device 64. This pointer icontypically takes the form of a small arrow, a pointing hand, or the like.The sensor 75 senses the movement of the object 76 and communicates themovement to the local microprocessor 68 through the sensor interface 72.The local microprocessor 68 then communicates through communicationcoupling 88 to the microprocessor 36 to cause the microprocessor 36 tocreate a corresponding movement of the pointer icon on the imagedisplayed upon the visual display 64. In some embodiments, the sensors74 can communicate directly to microprocessor 36 without the use oflocal microprocessor 68. The user can also create other input, such as a“button click,” through the other input 78 which are communicated to themicroprocessor 36 by the local microprocessor 68 or directly, e.g.,using a game port.

If the pointer icon on the display device 64 is at a position (or time)that correlates to a desired force feedback to the user 52, themicroprocessor 36 sends a force feedback command to the localmicroprocessor 68 over the serial port connection 88. The localmicroprocessor 68 parses this force feedback command and sends signalsto the actuator interface 80 which causes the actuator 84 to createforces F on object 76, which are experienced by the user 52 as indicatedat 60. The safety switch 82, sometimes referred to as a “deadmanswitch”, blocks the signal from the actuator interface 80 if, forexample, the user 52 is no longer grasping the object 76. In this way,the user 52 can interact with the client machine 46 in a visual,auditory, and tactile fashion.

The hardware architecture described above is also described inco-pending U.S. Pat. No. 5,739,811, filed Nov. 28, 1995, the disclosureof which is incorporated herein by reference. The high level commandprotocol between the computer and the force feedback device is alsodescribed in U.S. Pat. No. 5,734,373, filed Dec. 1, 1995, the disclosureof which is incorporated herein by reference. Force feedback asimplemented in a graphical user interface is described in U.S. patentapplication Ser. No. 08/571,606, now U.S. Pat. No. 6,219,032, filed Dec.13, 1995, and incorporated herein by reference.

FIG. 4a is a perspective view of a force feedback mouse interface system90 of the present invention, capable of providing input from the user toa host computer based on the user's manipulation of the mouse andcapable of providing force feedback to the user of the mouse system inaccordance with the present invention. Mouse system 90 includesinterface device 50 that includes a user manipulatable object ormanipulandum 36 and an interface 100, and host (client) computer 48.

User manipulatable object (or “manipulandum”) 36, in the describedembodiment, is a mouse that is shaped so that a user's fingers or handmay comfortably grasp the object and move it in the provided degrees offreedom in physical space. For example, a user can move mouse 36 tocorrespondingly move a computer generated graphical object, such as acursor or other image, in a graphical environment provided by computer48. The available degrees of freedom in which mouse 36 can be moved aredetermined from the interface 100, described below. In addition, mouse36 preferably includes one or more buttons 102 to allow the user toprovide additional commands to the computer system.

It will be appreciated that a great number of other types of usermanipulable objects can be used with the method and apparatus of thepresent invention in place of or in addition to mouse 36. For example,such objects may include a sphere, as trackball, a puck, a joystick, aknob, a wheel, a dial, cubical-, cylindrical-, or other-shaped handgrips, a fingertip receptacle for receiving a finger or a stylus, a flatplanar surface like a plastic card having a rubberized, contoured,and/or bumpy surface, a handheld remote control used for controlling webpages or other devices, pool cue, or other physical objects.

Interface 100 is provided in housing 101 and interfaces mechanical andelectrical input and output between the mouse 36 and host computer 48implementing the application program, such as a GUI, simulation or gameenvironment. Interface 100 provides one or more degrees of freedom tomouse 36; in the preferred embodiment, two linear, planar degrees offreedom are provided to the mouse, as shown by arrows 104. In otherembodiments, greater or fewer degrees of freedom can be provided, aswell as rotary degrees of freedom. For many applications, mouse 36 needonly be moved in a very small workspace area.

The interface 100 provides position information to computer 48 via bus112, such as a Universal Serial Bus (USB). In addition, computer 48and/or interface 100 provide force feedback signals to actuators coupledto interface 100, and the actuators generate forces on members of themechanical portion of the interface 100 to provide forces on mouse 36 inprovided or desired degrees of freedom. The user experiences the forcesgenerated on the mouse 36 as realistic simulations of force sensationssuch as jolts, springs, textures, enclosures, circles, ellipses, grids,vibrations, “barrier” forces, and the like. The electronic portion ofinterface 100 may couple the mechanical portion of the interface to thehost computer 48. Interface 100 preferably includes a localmicroprocessor 68 as described above, Mouse 36 is preferably supportedand suspended above a grounded pad 110 by the mechanical portion ofinterface 100.

Computer 48 is preferably a personal or other computer, workstation, orgame console as described above. Computer 48 preferably implements oneor more application programs (“applications”) with which a user isinteracting via mouse 36 and other peripherals, if appropriate, andwhich can include force feedback functionality. For example, one of theapplication programs is preferably a Web browser that implements HTML orVRML instructions. Herein, computer 48 may be referred as displaying“graphical objects” or “objects.” These objects are not physicalobjects, but are logical software unit collections of data and/orprocedures that may be displayed as images by computer 48 on displayscreen 64, as is well known to those skilled in the art. Display device64 can be included in host computer 48 and can be a standard displayscreen (LCD, CRT, etc.), 3-D goggles, or any other visual output device.

As shown in FIG. 4a, the host computer may have its own “screen frame”118 (or host frame) which is displayed on the display screen 64. Incontrast, the mouse 36 has its own “device frame” (or local frame) 120in which the mouse 36 is moved. In a position control paradigm, theposition (or change in position) of a user-controlled graphical object,such as a cursor, in host frame 118 corresponds to a position (or changein position) of the mouse 36 in the local frame 120.

FIG. 4b is a perspective view of a preferred embodiment of themechanical portion 108 of mouse device 50. Mechanical linkage 130provides support for mouse 36 and couples the mouse to a groundedsurface 124, such as a tabletop or other support. Linkage 130 is, in thedescribed embodiment, a 5-member (or “5-bar”) linkage. Fewer or greaternumbers of members in the linkage can be provided in alternateembodiments.

Ground member 132 of the linkage 130 is a base for the support of thelinkage and is coupled to or resting on a ground surface 124. Themembers of linkage 130 are rotatably coupled to one another through theuse of rotatable pivots or bearing assemblies, all referred to as“bearings” herein. Base member 134 is rotatably coupled to ground member132 by a grounded bearing 142 and can rotate about an axis A. Linkmember 136 is rotatably coupled to base member 134 by bearing 144 andcan rotate about a floating axis B, and base member 138 is rotatablycoupled to ground member 132 by bearing 142 and can rotate about axis A.Link member 140 is rotatably coupled to base member 138 by bearing 146and can rotate about floating axis C, and link member 140 is alsorotatably coupled to link member 136 by bearing 148 such that linkmember 140 and link member 136 may rotate relative to each other aboutfloating axis D. Mouse 36 is coupled to link members 136 and 140 byrotary bearing 148 and may rotate at least partially about axis D.

Transducer system 150 is used to sense the position of mouse 36 in itsworkspace and to generate forces on the mouse 36. Transducer system 150preferably includes sensors 152 and actuators 154. The sensors 152collectively sense the movement of the mouse 36 in the provided degreesof freedom and send appropriate signals to the electronic portion ofinterface 100. Sensor 152 a senses movement of link member 138 aboutaxis A, and sensor 152 b senses movement of base member 134 about axisA. These sensed positions about axis A allow the determination of theposition of mouse 36 using known constants such as the lengths of themembers of linkage 130 and using well-known coordinate transformations.Sensors 152 are, in the described embodiment, grounded optical encodersthat sense the intermittent blockage of an emitted beam. A groundedemitter/detector portion 156 includes an emitter that emits a beam whichis detected by a grounded detector. A moving encoder disk portion or“arc” 158 is provided at the end of members 134 and 138 which each blockthe beam for the respective sensor in predetermined spatial incrementsand allows a processor to determine the position (and velocity) of thearc 158 and thus the members 134 and 138 by counting the spatialincrements.

Transducer system 150 also preferably includes actuators 154 to transmitforces to mouse 36 in space, i.e., in two (or more) degrees of freedomof the user object. The bottom housing plate 157 of actuator 154 a isrigidly coupled to ground member 132 (or grounded surface 124) whichincludes, e.g. a magnet, and a moving portion of actuator 154 a (e.g. awire coil) is integrated into the base member 134. The actuator 154 atransmits rotational forces to base member 134 about axis A. The housing157 of the grounded portion of actuator 154 b is coupled to groundmember 132 or ground surface 124 through the grounded housing ofactuator 154 b, and a moving portion (e.g. a coil) of actuator 154 b isintegrated into base member 138. Actuator 154 b transmits rotationalforces to link member 138 about axis A. The combination of theserotational forces about axis A allows forces to be transmitted to mouse36 in all directions in the planar workspace provided by linkage 130through the rotational interaction of the members of linkage 130. Theoperation of the electromagnetic actuators 154 is described in greaterdetail in U.S. Pat. Nos. 6,100,874 and 6,166,723. In other embodiments,other types of actuators, such as electrical DC motors, can be used. Adifferent embodiment of a force feedback device can include flexuremembers to allow movement in provided degrees of freedom.

In FIG. 5, a conceptual representation of the network system 10 withforce feedback includes a server machine 18, a client machine 14provided with a force feedback device 24, and one or more additionalclient machines 16, each of which may be provided with additional forcefeedback devices 26. As noted in this figure, the server machine is acomputer or “processor” running, for example, the TCP/IP server softwareand is which is connected to the Internet. The client machine 14includes a computer or “processor” running Internet browser software andforce feedback driver software. The processor of the client machine isconnected to the Internet and to the force feedback device 24. The forcefeedback device 24 has sensors and actuators so that it can trackmovement of the user manipulatable object, monitor for button pressesand/or other ancillary input devices, and provide output force feedbacksensations. The force feedback device 24 sends object trackinginformation to the client machine, and receives force feedback commandsfrom the client machine 14. The “additional client”, such as clientmachine 16, also includes computers or “processors” running Internetbrowser software and force feedback driver software. The processors ofthese additional clients are also connected to the Internet and areconnected to force feedback devices associated with that client.

As noted in FIG. 5, a client machine 14 can send a data request to theserver machine 18 and, in return, receive a web page including HTMLinstructions and/or other instructions for implementing a web page, e.g.Dynamic HTML instructions, JavaScript instructions, Java or ActiveXcode, an embedded plug-in reference (such as the “IFF” extensiondescribed below), etc. For example, if an embedded IFF reference isused, the server must also have a modified configuration file which letsit know that IFF is a valid MIME type. This modified file would be aSRM.CONF or other .CONF file, as will be appreciated by those skilled inthe art. The client machine 14 then sends force feedback commands to theforce feedback device 24 and receives tracking and button data from theforce feedback device 24. Client machine 16 can likewise send a datarequest to the server machine 18 and receive an HTML file and/or otherweb page data or other document or collection of data. The clientmachine 16 can then interact with the force feedback device 26 bysending force feedback commands to the device 26 and by receivingtracking and button data from the force feedback device 26.

In another embodiment, a force-related application or control programrunning on the client machine 16 can check if any peripherals includingforce feedback functionality are connected to the client machine. Forexample, after downloading a web page having force effects, the clientmachine can first check whether a force feedback mouse is connected. Ifno mouse is connected, the client can check whether another forcefeedback peripheral, such as a force feedback joystick, is connected. Ifso, the client can enable the joystick to control the position of thecursor within the web page so that the user is able to experience theforce effects associated with web page objects. Other force feedbackperipherals can also be enabled for cursor control in a web page. Theforce-related application can perform the necessary adjustments toreceive input from an absolute input device or a relative input device,as necessary, or from a “cursor control device” or a “gaming device”,etc.

In addition to communicating with the server machine, the clientmachines can communicate directly with each other over the Internetusing an Internet communication protocol. For example, client machine 14can communicate with client machine 16 through a TCP/IP connection. Thisis accomplished making the URL of the client machine 16 known to theclient machine 14, and vice versa. In this fashion, direct communicationbetween client machines can be accomplished without involving the servermachine 18. These connections can send force feedback information andother information to the other client machine. For example, a process onthe client machine 16 can send force feedback infonnation over a TCP/IPInternet connection to the client machine 14, which will then generate aforce feedback command to the force feedback device 24. When the userreacts to the force feedback at force feedback device 24, thisinformation can be sent from client machine 14 to client machine 16 toprovide force feedback to the user on force feedback device 26.

As is well known to those skilled in the art, a client machine normallyconnects to another computer, such as a server or other client, over theWeb by sending a connection request to the “host” of the desired URL.The host, in this example, is a server machine 18 and the desired URL isthe URL of the desired web page residing on the server machine 18.Alternatively, the desired web page can reside on another server orresource and be retrieved by server machine 18. In response to theconnection request of step 150, the server machine 18 sends the HTMLfile representing the web page over the Internet to be received by theclient machine. The HTML file includes a number of “components” whichare typically commands, command fragments, instructions, and data whichpermit the display of the web page and other functionality. HTMLcomponents are parsed and interpreted (processed) as they are received,e.g., even before the entire HTML file is received at the clientmachine. Alternatively, the entire HTML file can be received before theprocessing begins.

A HTML file typically includes a number of “components” which are parsedand interpreted as previously described. For example, an HTML filebegins with a <HTML> command or “tag” to indicate the start of the HTMLfile, and a <BODY> tag to indicate that the body of the HTML file isbeginning. Then, an arbitrary number of HTML commands are provided to,for example, display images of the web page on the video display of theclient machine. The body of the HTML file is terminated by the </BODY>command, and the end of the HTML file is indicated with the </HTML>command, i.e. this command is the “eof” command of the HTML file.

Force Effects in Web pages

There are many types of touch interaction that can be achieved in Webpages. The force sensation resulting from this interaction is knownherein as a force effect or an “effect.” Two main classes of tactileinteraction with Web content described herein are “generic effects” and“authored effects,” as described in greater detail below. These effectsare applied to web page objects in the most common implementation.Herein, “web page objects” or “web objects” are any graphical objects(or regions) appearing on a web page, such as images, text, buttons, orother standardized or custom objects. In some cases, a web page objectmay not actually appear on the web page when displayed, but may have aneffect on how the web page is visually displayed or how force effectsare output on the force feedback interface device.

Generic effects and authored effects are preferably composed from abasic set of stock force effects. The stock effects include vectorforces, vibrations, springs, textures, and others, as described in U.S.Pat. Nos. 5,825,308; 6,219,032; 5,959,613; 6,147,674; and 6,078,308, allincorporated by reference herein. Effects of differing complexity can beprovided as stock effects; for example, a primitive effect such as asimple vector force to be output in a specified direction at a specifiedmagnitude can be provided, or a more complex effect that includesmultiple primitive effects can be provided. One particularlysignificant, more complex effect is the enclosure. An enclosure is a setof forces that occur only when the cursor is in or near a geometricallybounded (“enclosed”) area of the screen. Enclosures can be associatedwith forces at their borders to attract the cursor to the inside of thebounded area, keep the cursor outside the bounded area, or attract thecursor to the border surrounding the bounded area. The enclosure maytake a variety of shapes, for example rectangular or elliptical, and mayalso be associated with one or more other force effects when the cursoror pointer is positioned inside the enclosure. Examples of force effectsthat can be provided and programmed are specified in the FEELitApplication Programming Interface (API) from Immersion Corporation ofSan Jose, Calif., detailed in patent application Ser. No. 08/970,953,filed Nov. 14, 1997, and incorporated by reference herein.

For example, a hyperlink (“link”) is an input-sensitive area displayedon a web page which, when selected by the user, causes a “link action”in the web browser. Such link actions can include retrieving a differentweb page over the Web whose address is specified by the link and/ordisplaying that web page, causing a different section of the web pagecurrently in memory to be displayed, retrieving data for an animation orother modification of the current web page, sending data to a serverover the web, or other action either locally and/or involving datatransfer over the WWW. Typically, one or more web objects on a web pageare defined as hyperlinks, such as text, an image, an animated image, anicon, or other object; such an object is referred to as a “link object”herein. The user typically selects a link by moving the mouse pointerover the link object and providing a command gesture such as clicking amouse button. Alternatively, links might automatically be selected whenthe mouse pointer is moved over the link object, i.e., the interactionof pointer and link object is the command gesture in such a case.

An enclosure can be defined as the visual boundaries of a hyperlinkobject (enclosures can also be defined as a portion of an object or as aportion of the background to the web page). The forces of the enclosurecan be used to “snap” the pointer to that link object, i.e., assist thepointer to move onto or “acquire” that link object. To the user, thismight feel as if the link has a magnetic attraction. A text hyperlink,for example, is no longer purely discerned by the user by the text'sappearance (usually color and/or underlining), but is also discerned byits feel. Generally, hyperlinks are some of the more interestingfeatures of a Web page and are intended to stand out from the other webpage content to inform the user that other content can be accessed fromthe displayed web page, e.g. hyperlinks are the areas on the web pagewhere the location of the mouse pointer can cause a link event, whileother areas on the web page may have no effect based on input from amouse. The snap enclosure effect physically grabs the user's attentionat a link using forces until the user pushes the user manipulatableobject (such as a mouse) hard away from the link object. Instead ofinteracting with unresponsive text, the enclosures constrain the user tothe most important content on the web page that is responsive to inputfrom the mouse (e.g. a command gesture such as a button click).

In addition, some objects on the web page may be defined as a hyperlink,but there is no visual evidence to the user that these objects are linksand can cause a link action such as calling up a different web page. Forinstance, many graphical images are also hyperlinks, but this fact isnot discernible visually; previously, the user had to move the pointerover the image, which would cause the appearance of the mouse pointer tochange if the image were a link object (or cause a change in otherdisplayed information). By providing enclosure effects on hyperlinks,tactile cues are offered instead of or in addition to visual cues; thesetactile cues are consistent with the user interface so the user easilyunderstands their meaning. The user can more quickly determine whichobjects on a web page are links by using the tactile sense compared withthe visual sense. Adding feel to web pages increases both the user'sefficiency and the quality of their experience. Some studies have shownan 88% improvement in mouse targeting performance with over 1000 testsubjects.

Other objects on a web page besides hyperlinks may also be associatedwith forces such as the enclosure to assist acquiring the object. Forexample, standard GUI graphical objects such as text boxes, inputbuttons, radio buttons, checkboxes, icons, are readily adaptable toenclosures. In addition, the user can be constrained by enclosures tothe input fields or the areas associated with frames. For example, aninput field where the user must enter data such as a name, a credit cardnumber, or click a drop down menu are prime areas which can beassociated with enclosure “snap to” forces. Similarly, frames havingparticular objects of interest can be associated with enclosure effects.“Frames” are those areas of a web page which can be accessed,manipulated, scrolled, etc. independently of other displayed areas(other frames) of the web page. For example, a frame on the left side ofa web page often is used to display the main subject areas of the webpage while the larger, main frame on the right displays the content of aselected one of those subject areas.

Finally, an extra dimension of user experience can be added by invokinga “pop” or jolt force effect when the user clicks on a buttons,checkboxes, or other graphical object. A pop is a time based sensationas opposed to, e.g., an enclosure implemented as a snap sensation, whichis a spatially based sensation.

FIG. 6 illustrates a web browser 200 such as Netscape Navigator in whichdisplayed web page objects are associated with forces. For example,these force effects can be authored force effects using any of theauthored effect embodiments described below. Control area 210 of thebrowser includes well known controls such as navigation buttons 212 and214 and URL address field 216. The display area 230 has been providedwith a number of web page objects to illustrate some of the concepts ofthe present invention. The force feedback device controls the positionof a pointer icon (cursor) 240 which can be caused to interact with thevarious web page objects.

As an example, when the force feedback device is manipulated by the userto cause the pointer icon 240 to move within a “texture” region 242,force feedback commands can be created for the force feedback device toprovide a desired “texture” to the force feedback device. For example,the texture can feel “rough” to the user by causing the force feedbackdevice to place forces on the user manipulatable object that emulate arough or bumpy surface. In a region 244, a viscosity or “liquid” forcefeedback can be provided. With this form of force feedback, as thepointer icon is moved through field 244, a viscous “drag” force isemulated on the user manipulatable object as if the user object weremoved through a thick liquid. In a region 246, inertial forces can befelt. Therefore, a pointer icon being moved through an “inertia” regionwould require relatively little or no force to move in a straight line,but would require greater forces to accelerate in a new direction or tobe stopped. The inertial force sensations can be applied to the usermanipulatable object and felt by the user.

In a “keep out” region 248, the pointer image is prevented from enteringthe region. This is accomplished by creating a repulsive force on theuser manipulatable object using a force feedback command to the forcefeedback device which prevents or inhibits the user from moving the usermanipulatable object in a direction of the region 248 when the pointericon 240 contacts the periphery of the region 248. In contrast, a“snap-in” region 250 will pull a pointer icon 240 to a center 252whenever the pointer icon engages the periphery of the snap-in region250 and apply a corresponding attractive force on the user manipulatableobject. A “spring” region 254 emulates a spring function such that apointer icon moving into the spring region “compresses” a spring, whichexerts a spring force on the user manipulatable object which opposes themovement of the pointer icon. A region 256 is a “Force To Left” regionwhere the pointer icon within the region 256 is forced to the left sideof the region and the user manipulatable object is forced in acorresponding direction as if influenced by some invisible magneticforce or gravitational force. A region 258 illustrates that regions canbe of any size or shape and that within a region different force effectscan be developed. In this example, within region 258 there is a texturecore 260 surrounded by a vibration ring 262. Therefore, as the pointericon 240 moves into the region 258, the user first experiences vibrationfrom the ring 262, and then experiences a texture as the pointer iconmoves within the core 260.

The exemplary force feedback web page of FIG. 6 is also provided withseveral force feedback buttons. In a first button 264, the placement ofthe pointer icon 240 over the button and the pressing of a mouse button(i.e., a switch) on the mouse 36 to create a “button click”, “buttondown”, or simply a “button event” input, will then cause a “buzz”command to be sent to the force feedback device. The buzz command would,for example, cause a vibration force on the user manipulatable object.Similarly, the selection of the “jolt” button 266 will cause a joltingforce (e.g., a short-duration pulse of force) to be provided at theforce feedback device, and the pressing of the “detent” button 268 willcause a “detent” to be created for the force feedback device. By“detent” it is meant that the user manipulatable object will becontrolled by the force feedback actuators such that it feels as if amechanical-type detent exists at the position that the usermanipulatable object was in when the detent button 268 was activated.

Generic Effects

Generic effects are force effects that are applied uniformly to allobjects in a received document having a particular type. For example, ina received web page, web page objects of standard types includehyperlinks, images, text, text entry fields, tables, headings, imagemaps, marquees, buttons, check boxes, radio buttons, drop-down menus, orother standard web page objects. Different generic effects can also beapplied to different characteristics of an object; for example, oneeffect is associated with bold text, while a different effect isassociated with text having the color blue. Generic effects are achievedby defining a mapping between object types and feel sensations, wherefeel sensations are uniformly associated with particular types of webpage objects and/or particular types of interactions (or “events”) withobjects. “Interactions” or “events” may include clicking a mouse buttonwhen the pointer is positioned over the object, moving the pointer ontoa link from above the link (e.g. as differing from moving the pointeronto the link from below the link), or any other cursor/objectinteraction defined by the developer or user.

In a preferences file or other storage of the client machine, particularforce effects are each mapped to a particular type of object provided ina web page (and/or GUI). Using generic effects, the same force effect isapplied to all graphical objects of a particular type. For example, allhyperlinks on a page can be assigned with an enclosure having the sameforce characteristics (e.g., forces of the same magnitude), although thearea enclosed by each enclosure on the page will vary with the size ofthe web page object with which it is associated.

A major advantage of using generic effects is that the force effects areimplemented exclusively on the client side of the network. Thus, theauthor/content developer of a web page does not need to specify anyforces or force characteristics of the objects in the web page codeitself. When the web page is received by the client computer, theforce-enabling code implemented by the client associates the forces fromthe preferences file (or default forces) to each graphical object andoutputs the forces as appropriate. This allows standard web pages thathave no force effects in their code to be implemented as a forcefeedback web page on the client end, which allows a force feedbackinterface device to be used immediately without requiring special webpages tailored for force output. In other embodiments, a web page caninclude force feedback information for authored effects (describedbelow), and generic effects can also be applied to web page objects nothaving any authored effects associated with them, or to overrideparticular authored effects as desired by the user of the client.

For example, the force magnitudes or directions associated with anenclosure are defined on the client computer. This enclosure is assignedto objects of a particular type, e.g. link objects. A differentenclosure with different associated forces can be assigned to anothertype of object, such as a (non-link) image or an animation. Thepreferred embodiment allows the user to adjust the effects to suit theirpreferences. For example, one way to allow a user to assign forces isanalogous to the approach applied to the color of a hyperlink object,where the web browser automatically assigns the hyperlink a color basedon the user's preference but web authors can override those defaultcolors as desired, e.g. to make the hyperlinks' color match the webpage's color scheme. The user preferably is able to adjust thecharacteristics of generic effects using a central control panel, e.g.,a control dialog box associated with their force feedback GUIpreferences, or a dialog box associated with the web browserpreferences.

In a different embodiment, different sets of generic effects can beprovided which are associated with different web pages or differentgroups of web pages. For example, the user can specify a first set ofgeneric effects in which snap enclosures are to be associated with linkobjects on the downloaded web page. This first set is to be associatedwith web pages of a particular type or origin, e.g. all web pagesreceived from a particular URL or domain name will have objects assignedto the first set of effects on the client computer. Likewise, a secondset of generic effects can be characterized by the user, e.g. assignsvibration forces to link objects. The second set of generic effects isassigned to web pages received from a different source or having adifferent characteristic. Alternatively, different generic effect setscan be provided by different web site providers, developers, companies.Thus, for example, Immersion Corporation can provide a generic effectset that the user can download and which will automatically be assignedto web pages that are downloaded from specified servers or locations onthe WWW. (The specified locations can be designated in the genericeffect set file, or otherwise). The web browser (or a plug-in or script)can check for these specified locations when a web page is downloaded.

Authored Effects

In contrast, authored effects are effects particular to a certain webpage that are specified or modified by a content developer or otherauthor and which can provide a more rich, custom experience than genericeffects can offer. For example, one or more particular graphical objectson a web page can be assigned a different, customized force effect by anauthor (the “author” can be the creator/developer of the web page or theuser on the client machine that receives the web page). Web page objectshaving the same type can be assigned different force effects. Authoredeffects can be used, for example, to intentionally correlate aparticular, unique force with an object of a web page. Typically, atleast part of the instructions governing the type and/or characteristicsof an authored effect are included in the code of the web page that isdownloaded by the client. Thus, standard non-force web pages do notinclude authored force effects (unless there is preprocessing done tothe web page before it is displayed by the client, as described ingreater detail below).

Any standard force effect can be an authored effect, e.g. those feelsensations supported by the FEELit API from Immersion Corp.Alternatively, authored effects can be partially or completelycustomized to be specific to a desired interaction or display. Oneexample of an authored effect is gravity toward a particular object. Agravity force is an attractive force that biases or actively attractsthe mouse in a direction that causes the mouse pointer to move toward adefined point of the desired object. For example, an author can usegravity to attract the mouse pointer towards advertisement image objectson a web page. Many web sites are sponsored or supported by advertisers;such sites usually include web pages that display graphical hyperlinksto a sponsor's web site. Although the advertisement visuals (animations,etc.) can attract the attention of a site's visitor, an authored effectcan physically attract the mouse toward the advertisement, forcing theuser to acknowledge the advertisement's existence. However, not all thegraphical images on the same web page may be desired to have a gravityeffect, so that generic effects may not be suitable in this situation.In other examples, authored effects can enable users to feel the textureof clothing at a commercial Web site by providing a texture effectassociated with a graphical clothing image, or to feel the forces of afalling ball at a physics education web site by providing a ball imagethat causes the desired forces when the pointer is moved in contact withthe ball image. Authored effects allow an author to provide a “theme” offeel sensations in accordance with a theme or subject of a particularweb page; for example, a vibration force effect can be associated withthe links of a web site that gives information about earthquakes.

Authored effects can be “dynamic,” i.e., the behavior of the effect canchange depending on the author's intent and a current state or otherstate of the Web page. In the gravitational advertisement example, thegravity can be turned off once the mouse cursor has reached theadvertisement. Thus, the user's experience of the site will not becompromised after the cursor has been guided to the advertisement. Thedynamic nature of these effects can be achieved either through scripting(e.g. using such languages as VBScript or JavaScript) or programming(e.g., using such programming constructs as Java applets and ActiveXcontrols). The ability to incorporate compelling authored effects in Websites is limited only by the developer's creativity.

Although authored effects can allow much more compelling forces to beprovided in web pages because they can be carefully correlated withimages, animations, and sounds. They have the disadvantages of typicallyrequiring specific force effect instructions in the web page that aredownloaded by the client and sometimes require specific force-enabledresources on the client end.

A different embodiment uses a form of authored effects to specifygeneric effects. For example, if an author does not want to includeauthored effects in a web page but wishes to specify one or more genericforce effects for object types on the web page, the author can includegeneric effect designations. For instance, a number of standardizedgeneric effects sets (as described above) can be provided on clientmachines, and each web page downloaded can specify in an HTML tag (orotherwise) which generic effect set (or a particular generic effectwithin a set) to use for the downloaded page. Thus, the author has, ineffect, authored the force effects by referring in the web page togeneric effects resident on the client. Or, the tag in the web page codecan refer to an official or customized generic effect set that the useris presumed to have previously installed on the client machine (and, ifthat customized effect set is not installed on the client, a defaultgeneric effect set can be used). The author can simply specify thatthere is a force enclosure on an object, but allow client defaults orpreferences to assign forces and/or force characteristics to theenclosure (e.g. to the walls and/or interior of the enclosure).

A user can experience both authored effects and generic effects in aparticular downloaded web page. For example, the user can set genericeffects to types of objects displayed in the web page, as defined by theuser's preferences stored on the client. There can also be authoredeffects in the web page that are specified in information downloadedwith the web page. Any authored effect that conflicts with a genericeffect can be set, by default, to override the conflicting genericeffect. This allows special authored effects to still be felt by theuser, yet also allows any other web page objects not assigned anyauthored effects to have the user's preferred generic effects assignedto them. As is analogous to the color of hyperlinks, the user on aclient machine can also preferably specify whether to override anyauthored forces of a downloaded web page with preferred generic effects;or the user can specify to use the authored effects or the genericeffects specified by the web page author (which may fit the theme of theweb page better than the user's choice of generic effects, for example).

In yet another embodiment of the present invention, either authoredeffects or generic effects can be implemented by continuous reception offorce feedback data over a network, e.g. similar to “streaming” videoand audio. For example, parameters to specify and characterize forcescan be included in streaming video and audio data to provide tactilesensations to the user of the client machine receiving the streamingdata. Tags or references to force effects already resident on thereceiving client machine can be included in the streaming data.Alternatively, all or most of the data defining the force effects andthe data required to implement the force effects can be included in thestreaming data.

Embodiments for Implementing Force Feedback over Networks

The following subsections detail several embodiments of software thatallow a user to feel web pages. These embodiments retrieve web pagedata, track user interaction with the web page, and output theappropriate force effects to the tactile display.

Proprietary Browser

An efficient embodiment to enable feel in web pages is to implement theforce effect functionality directly in the web browser, such as NetscapeCommunicator by Netscape Communications or Internet Explorer byMicrosoft Corp. A web browser parses Hypertext Markup Language (HTML)files retrieved over the Internet and visually displays the content ofthe web page. In the present embodiment, a browser may also track auser-controlled cursor's interactions with the displayed content andapply force effects accordingly. Since the web browser has intimateknowledge of the web content, it can perform these tracking operationsmore easily than many other methods can do so. This embodiment is mostfeasible for entities that have already written a Web browser (such asMicrosoft and Netscape). In addition, publicly available source code ofa web browser (e.g., Netscape Communicator) can be modified by adifferent party to incorporate force feedback functionality directlyinto the browser and re-released to the public.

This embodiment can apply generic effects to web page objects, which istypically used for standard web pages. However, HTML tags (commands) canalso be created which can define authored effects in a downloaded webpage and which the web browser can interpret and implement, similar tothe function of the plug-in software described below in the plug-inembodiment. The use of authored effects can alternatively be achieved inthis embodiment by providing Java applets or ActiveX controls in the webpage, as described in greater detail below.

Dynamic HTML

Dynamic HTML is a set of features currently in browsers such asMicrosoft Internet Explorer 4.0 that enable authors to dynamicallychange the rendering and content of an HTML document. Using DynamicHTML, a content developer or programmer can access the attributes of thegraphical objects of an HTML document (such as the object's position onthe web page and the object's HTML tag or type). In addition, eventmessages are generated when a user selects or interacts with the Webobjects (such as when a user clicks a button while the mouse pointer isover the object). The power of Dynamic HTML can be elicited, forexample, through the use of VBScript or JavaScript scripts embedded in aweb page or prograinmatically, e.g. in a web browser, through VisualBasic or C++, as is well known to those skilled in the art.

Dynamic HTML can be used to enable the feeling of generic effects in webpages. FIG. 7 presents a flow diagram of the present invention inaccordance with a first Dynamic HTML embodiment in which a separateapplication program runs in parallel with a web browser on a clientmachine. A user starts a separate application program on a clientmachine in step 284. In a step 286, the application launches a webbrowser on the client machine, and creates/maintains a connection to theweb browser. In step 290, this application then “hides” in thebackground of the operating system. After a web page in HTML has beenreceived and displayed by the web browser, the application checks for anevent notification in step 292. The application receives an eventnotification from the browser whenever a user moves the mouse (and/orwhen another predefined event occurs). When the application receivesthis notification, it can ascertain in step 294 whether the mousepointer is touching any relevant object (e.g., a web page object havingforces associated with it) on the web page through the use of DynamicHTML functions. If no event notification is received in step 292, theprocess continues to check for a notification until the user quits theweb browser or the separate application in step 300, at which point theconnection to the web browser is destroyed in step 302.

One example of determining if a relevant object is being touched isprovided in the pseudocode below. Dynamic HTML is hierarchical innature, where characteristics of an object may be provided in parentobjects to the object. Since Dynamic HTML is hierarchical, the elementpassed immediately to a requesting application or function may not be ofinterest, but something in that element's hierarchy may be of interest.For example, only bold text can be associated with a force effect. Theapplication thus must check whether the current object that is “touched”by the pointer is bold text. The below function checks this hierarchy.Given a Dynamic HTML object, called “elem”, this function will determinewhether the object is something to which a force-effect is to be applied(i.e., it is “relevant”). If a text object is passed to this function,this function will search the hierarchy of the text object looking for a“B” tagname that indicates the text is bold. It will return null if theobject is not relevant, otherwise it will return the relevant tagname.

// This is peeudo-code for the TouchingRelevantObject function. TAGTouchingRelevantObject ( HTMLElement elem ) { // Is this element valid?// This will happen if we’ve reached the end of the recursion. if ( elemIS null ) return false; if ( elem.tagName IS_IN A_SET_OF_TOUCHABLE_TAGS) return elem.tagName; else return TouchingRelevantObject(elem.parentElement ); }

If the mouse pointer happens to be touching a relevant object, i.e. anobject associated with one or more force effects (such as a hyperlink),the application calculates the screen coordinates of the object in step296 using the dynamic HTML functions. The application then generates theappropriate force effect in step 298, such as an enclosure, on the usermanipulatable object of the interface device. The force effect can begenerated by using high level commands sent to the microprocessor of theinterface device 50, or can be generated by direct low level commandsfrom the host application.

One difficulty of this approach lies in the determination of screencoordinates of the relevant object. The difference between thecoordinate frame of the mouse and the coordinate frame of the DynamicHTML object requires that one of the coordinate frames be transformed tobe operable with the other. The mouse pointer position and effects suchas enclosures are described in screen coordinates. Dynamic HTML objectssuch as hyperlinks are described in coordinates relative to other Webpage objects; to derive the absolute screen coordinates of a DynamicHTML object, several coordinate frame transformations may have to beapplied based on the returned Dynamic HTML object. An example of thehierarchy of objects and the transformations required to achieve screenframe coordinates are illustrated in FIG. 8, which shows coordinateframes in Dynamic HTML. The frames may include a screen frame (absoluteframe) 310, a window frame 312, a web browser client area frame 314(which is also HTML frame #1), an HTML frame #2 316, and a hyperlinkframe 318. A particular frame is specified by its leftmost and topmostcoordinates, relative to the coordinate frame containing the particularframe. The hierarchy leading from the hyperlink frame to the screenframe is the reverse order of the frames as listed above, starting withthe hyperlink frame and traversing to the screen frame.

As currently implemented, it is not possible to compute the absolutescreen coordinates of a Dynamic HTML object by traversing itstransformation hierarchy because the farthest traversal is to the clientwindow frame; there are no mechanisms available for transforming thisclient window frame to the screen frame. Also, traversing this hierarchycan be computationally expensive, relative to the degree ofresponsiveness that is required in a force-feedback system. If a usermoves over a hyperlink object, the enclosure should snap immediately;any delays severely degrade the user experience.

The preferred way to resolve this problem is simple, elegant, and fast.Dynamic HTML returns an “event object” when the application is notifiedof a mouse interaction; this event object allows the program to querythe mouse position in both relative and screen coordinates. These valuescan be used to ascertain a Dynamic HTML object's transformation betweenrelative and screen coordinates. For example, the relevant “event”object fields that return the desired values are shown below:

offsetX—event X coordinate relative to the srcElement's container'sframe

offsetY—event Y coordinate relative to the srcElement's container'sframe

screenX—event X coordinate relative to physical screen size

screenY—event Y coordinate relative to physical screen size

srcElement—the object that cause the event

The relevant DHTML object fields are provided below:

offsetLeft—X coordinate of object's left edge, relative to itscontainer's frame

offsetTop—Y coordinate of object's top edge, relative to its container'sframe

These values indicate the coordinates of the web page object (also knownas an “element” of the web page) with reference to the object's“container”, which is the frame that encloses the object (such as theweb browser). Once these values are obtained, the absolute screencoordinates of the object can be determined as follows:

objectScreenX=event.screen.X−(event.offsetX−event.scrElement.offsetLeft)

objectScreenY=event.screen.Y−(event offsetY−event.scrElement.offsetTop)

These values are illustrated in FIG. 9.

Since generic force effects are often used with normal HTML code havingno force effects included therein, the Dynamic HTML is preferably onlyimplemented by the browser or other program providing the genericeffects. For example, after the HTML document is received, it isprocessed into Dynamic HTML by the browser or separate application onthe client so that web object properties can be obtained and genericforces applied. Alternatively, the web page can be provided as DynamicHTML instructions and downloaded in that form.

A second implementation of integrating generic force effects withDynamic HTML can be used in which the web browser itself has the abilityto check user interactions and apply force effects. As shown in a method320 of FIG. 10, the user again starts an application program on theclient machine in step 322. In step 324, this application opens abrowser window in the foreground that hosts navigation buttons and abrowser “engine”, e.g., an MSHTML object. MSHTML is the rendering engineused by Microsoft's Internet Explorer and is also available for allapplication developers to use. The MSHTML object performs the parsingand rendering of HTML content. To allow users to feel the displayedobjects of a web page, the application program then applies themethodology described above in FIG. 7 with the MSHTML object to directlydetermine (by calculating screen coordinates) when the mouse pointer ispositioned in relation to a relevant web object (in step 326) such thatscreen coordinates are calculated in step 328 and forces are to beoutput to cause the force feedback interface device to output theappropriate forces in step 330. The process quits via step 332 when thepointer no longer touches a relevant object.

The two implementations described above are summarized diagrammaticallyin FIGS. 11a and 11 b. In a first implementation of FIG. 11a, aproprietary application includes a connection to Internet Explorer,Netscape Navigator, or other available, standard web browserapplication. The browser 340 displays web page content 342. Theproprietary application 344 also runs simultaneously (e.g. multitasks)which is used to control forces for the web page content. The browser340 provides event notifications to the application 344 indicating, forexample, when the mouse pointer touches a relevant object. Upon eventnotification, the application 344 verifies if the element is touchable;if so, the application queries the browser 340 for element informationsuch as the element frame in screen coordinates. The proprietaryapplication 344 determines and outputs commands for generic forceeffects to the API (such as Feelit API) for the mouse or othermanipulandum, which are output as force sensations correlated with theweb page content displayed by the web browser 340.

In a second implementation of FIG. 11b, a proprietary application 348hosts MSHTML as described above, and does not connect to a standard,available web browser. This implementation uses the samenotification/query mechanism used in the first implementation.

The advantage of the first implementation is that a standard, well knownweb browser is used as the interface to the web page, which is familiarto users. Also, users can easily take advantage of web browser upgradesand improvements as provided by the browser company. Unfortunately, theconnection between web browser and the feel-enabling application programcan be tenuous. The connection can be prone to many bugs and may beslow, and may not have the responsiveness required by a force-feedbackapplication. On the other hand, in the second implementation, there isan extremely tight coupling between the browser engine (e.g. MSHTMLobject) and the feel-enabling application because they are in the sameprocess space. Thus, event notification in this implementation is robustand fast. The drawback is that users must use an interface that is notas full-featured as a proprietary web browser nor as familiar. They areidentical in terms of the display of web content, but a proprietary webbrowser typically has many more features and functionality formanipulating, editing, navigating among web pages, such as “Favorites”and “History” options to provide easier browsing.

The Dynamic HTML embodiment, when used without other tools, can onlyapply generic effects. In order to incorporate authored effects, it mustbe used in combination with either Java applets or ActiveX controls,both described below.

One example of providing generic effects to a web page using dynamicHTML is provided as source code in APPENDIX A. This source codeprovides, for example, snap enclosures on links.

Active Accessibility

Active Accessibility is a suite of programming libraries and computerprograms created by Microsoft Corporation to make the Windows operatingsystem more accessible to the disabled. One feature of ActiveAccessibility is that it can invoke third-party functions, known ashooks, when a user interacts in predefined ways with the graphical userinterface (GUI). For example, an application developer can create afunction that is called when the user moves the mouse or opens anywindow.

Using Active Accessibility to enable generic force effects for Web pageobjects involves the creation of a program that provides ActiveAccessibility hooks, including conditions which are to be checked andwhich functions to call when a condition is met. For example, theprogram runs on the client machine external to a web browser and ishooked into mouse moves. When the mouse move hook is invoked, theprogram instructs Active Accessibility to call one or more functionswhich request infonnation from Active Accessibility to determinecharacteristics of the web page object that the mouse pointer ispositioned over (if any), such as position coordinates, the type ofobject at those coordinates, the name of the object, othercharacteristics or properties, if the object is a link, etc. If theobject is associated with one or more force effects, the appropriatecalculations and force effects are applied. The limitation of thisembodiment is that the web browser must support Active Accessibility.Currently only one Web browser does so: Microsoft Internet Explorer.

This embodiment can be combined with a GUI force feedback controlapplication (such as the FEELit Desktop from Immersion Corporation) andused to enable generic effects in all browsers that support ActiveAccessibility. As with the proprietary browser and Dynamic HTMLembodiments, this embodiment can be combined with the Java applet orActiveX embodiments to enable authored effects.

Java Applet

Java is a popular programming language for web page development. Anapplet is a program written in Java that can execute in a web page whilethe web browser is the active application of the operating system on theclient, as is well known to those skilled in the art. A web pagedeveloper can create applets that make use of force feedbackfunctionality, e.g. using the FEELit API from Immersion Corporation.These applets can be similar to any program that can normally be createdwith such force feedback functionality, but can execute “inside” a webpage. An applet typically has a defined area on the web page in which itexecutes to display visual information or perform other tasks. Forexample, a developer working on a physics education web page can createan applet that allows a user to feel the force field between two virtualmagnets displayed on a web page; the user can interactively move themagnets using the mouse 36 and feel how the force field changes. Thefeel interactions may be limited to the active area in which the appletexecutes. However, the fact that Java applets are actually dynamicprograms, rather than static content, allows an endless range ofpossibilities for touch interaction with web pages.

There is one barrier to implementing this embodiment, but it is easilyovercome. The Java language was created to be platform independent andprovide strong security. A force feedback instruction set such as theFEELit API from Immersion Corp. may violate security restrictionsinherent in Java. However, Java has mechanisms for incorporating native(platform-specific) methods into Java applets. These mechanisms can beused to create a Java-compatible version of the FEELit API, allowingJava programmers to easily add force effects to their applets.

This embodiment can support authored effects and may be used inconjunction with the embodiments described above that support genericeffects, if desired.

ActiveX Control

The final embodiment of incorporating feel into web pages describedherein is the use of ActiveX controls implemented within the ActiveXlanguage by Microsoft Corporation. ActiveX controls are programmableobjects that can be “embedded” into (i.e. referenced and used by) webpages, as is well known to those skilled in the art. These objects canbe written in any language. The primary purpose of ActiveX controls isto add functionality to a web page that would normally be difficult, oreven impossible, using HTML and scripting languages. For example, a“graphing” ActiveX control can take spreadsheet data, display a graph ofit on a web page, and let the user manipulate it in real-time, e.g. byselecting a point on the graph and moving it, displaying a portion ofthe graph in greater detail, etc. Another feature of ActiveX controls isthat they can be scripted. This means that a content developer can embedan ActiveX control in a Web page and control it with a scriptinglanguage. For example, a JavaScript text box can be scripted to changethe title of the graphing control's title bar. Another way to performthis feat is to create a Java applet; however, the advantage of ActiveXcontrols is that they can be coded in any language. This advantage isalso potentially a drawback because a particular ActiveX control willonly work on one platform, whereas a particular Java applet will work onany platform.

There are two fundamental ways in which to use ActiveX controls to addfeel to web pages. The first way is to create an ActiveX control thatcontains dynamic visual and force content. Thus, each ActiveX controlembedded in a web page would have its own predefined force effects, andonly be able to perform those effects. This approach is similar to thatof the Java applet, except that the implementation language isdifferent. In particular, as currently implemented, the force content ofan ActiveX control is limited to the active area of the ActiveX control,just like a Java applet. This approach is faster than the second“force-only” approach since no scripting in HTML is required.

A second, more versatile, and preferred approach is to create a“force-only” ActiveX control that is solely responsible for generatingdifferent force effects (alternatively, a small number of ActiveXcontrols can be provided to control forces). This ActiveX control isembedded in (referenced by) the web page file and can be commanded withscripting instructions to create, modify, and trigger force effects.This scripting functionality allows content developers to easily tap thecapabilities of the ActiveX control without needing to do any customprogramming. For example, the force-only ActiveX control can be calledby JavaScript or HTML instructions in the web page with a command toperform a particular force effect. The ActiveX control would thencontrol the force feedback device to output the commanded forceeffect(s). For example, one way to call the ActiveX control“FeelControl” with a script in the HTML code of the web page is asfollows:

// This HTHL code line demonstrates how to make an image vibrate // whenclicking on it. // This force-only ActiveX control has an ID ofFeelcontrol and can // implement the force effect: // Vibration( longperiod, long duration ) // period -- in milliseconds // duration -- inmilliseconds // <IMG src=“myPicture.gif” onmousedown=“FeelControl.Vibration( 100, 1000 )”>

A central authority, company, organization, etc. can develop and refinethe force-only ActiveX control and allow web page authors to incorporatethe force-only control into their web pages. For example, a web pageauthor can embed a reference to the central web site to download theforce-only ActiveX control. The ActiveX control can be downloaded to theclient with the web page if the control is not already resident on theclient i.e., once the control has been downloaded to a client, it doesnot need to be downloaded again until the control has been updated to anew version or otherwise changed by the central authority. This isbecause the control is resident on client storage such as a memory cacheor hard disk after it has been downloaded the first time and can beaccessed locally by other web pages that embed it instead of downloadingit. With Such a control, a web page author can access the fullfunctionality of a force feedback command set such as the FEELit APIfrom Immersion Corporation using only the ActiveX control and ascripting language. Coupling this with Dynamic HTML yields a powerfultool for authoring feel into web pages.

To illustrate the usefulness of the second approach, several example webpages are illustrated in FIGS. 12-15. The web page shown in FIG. 12includes several images that represent tactile experiences, where theuser is able to feel a force associated with the image by moving thepointer over the image and/or maintaining the pointer on the image. Forexample, a user can move and maintain the mouse pointer over image 362to feel the force of an engine starting up and revving. Image 364provides a texture feel as if fingers were moved over tennis racketstrings. Image 366 provides a sequential jolt and vibration similar tothe sound effect of a laser. Image 368 provides an elliptical contourforce obstruction to allow the user to feel the edge of the depictedcrater. Image 370 provides a “slippery ice” sensation, where the mouseis assisted with forces to make it difficult to maintain the pointer onthe image. Finally, image 372 provides a texture similar to feeling acorduroy cloth when the mouse pointer is moved over it. The ActiveXcontrol commands a texture force to be output on the mouse so that theuser can feel it. An example of source code to implement these forceswith a force-only ActiveX control is provided in APPENDIX B. Thisexample uses hard-coded force effects; in other embodiments, forceeffects stored in separate files can be used with the ActiveX control.

The web page 380 shown in FIG. 13a and web page 387 in FIG. 13b areinteractive, dynamic multimedia demonstrations with feel. In web page380, a user may compress any of springs 382 a, 382 b, and 382 c usingthe mouse pointer by pushing the mouse pointer down on the end lines 386of the springs. A spring force is output on the mouse to resist thecompression motion. In web page 387, the user may similarly push themouse pointer against end lines 388 to compress the ball 384. If theuser compresses the ball 384 too much, it “pops” both visually andhaptically, i.e. a picture of a punctured, flattened ball is displayedand the user feels a jolt of the ball popping. The user can then returnthe ball to its original state by pressing button 389.

The web page 390 shown in FIG. 14 allows a user to move a basketball 392around a field or “court” by “pushing” the ball with the pointer 394.The mouse pointer is prevented from moving inside the ball 392 and abarrier force is output on the physical mouse when the pointer movesagainst any side of the ball, so that the user can feel the outline ofthe ball. In addition, the mouse movement against the ball causes theball to move in the same direction as the pointer, as if pushed by thepointer. The mouse will feel the force of motion of the ball if thepointer is moved in front of the moving ball.

Finally, the web page if FIG. 15 illustrates a dynamic force simulationincluding a moveable block or “cart” 396 with an upside-down pendulum398 extending from it on a pivotable rod 399. The pendulum is biased bygravity to pivot around its connection point to the cart. By moving thecart under the moving pendulum by pushing the cart left and right withthe mouse, a user can try to keep the pendulum from falling underneaththe cart. As the pendulum and cart moves, appropriate forces are outputon the mouse 36 depending on the motion/position of the pendulum andcart and accounting for a simulated weight of pendulum and cart. Thiscan serve as either a game, a physics lesson, or both. Thesedemonstrations only begin to hint at what is possible using webscripting languages and ActiveX controls. An example of source code usedto implement the dynamic force examples of FIGS. 13a, 13 b, 14, and 15is provided in APPENDIX C.

Both Java applets and ActiveX controls of this embodiment only supportauthored effects. However, it is possible to emulate generic effectsusing the second approach. Mechanisms for doing this are described inthe next section.

Implementing Generic Effects Using ActiveX Controls

The embodiments described above that support generic effects are theProprietary Browser, Active Accessibility, and Dynamic HTML embodiments.If none of these embodiments are available, another option may be used.Using mechanisms available in a force-only ActiveX control (describedabove), generic effects can be added to existing web pages as authoredeffects, with no need for the author of the web page to add the forceeffects manually.

The first method of achieving this is to pre-process an incoming HTMLfile. Whenever the web browser receives an HTML file, a separate programcan process it before it is parsed and displayed. This processing stepwould modify the HTML file, embedding the force-only ActiveX control,adding JavaScript code to facilitate force-effect scripting, andutilizing Dynamic HTML to assign effects to web page objects such ashyperlink objects. The types of effects added to the web page can bedetermined from a user-configurable preferences set or control panel,just as normal generic effects would be. For example, the userpreviously designated which types of force effects would be associatedwith which types of web page objects, and these preferences are readfrom a stored file or other computer-readable medium (or defaultmappings are used). The end result would be the experience of genericeffects although they are actually and transparently inserted into theweb page as authored effects by an external program.

The major drawback to this approach is that it is not seamless. Inparticular, it is difficult to assign a program to process an incomingHTML file. Few browsers, if any, support this functionality. Inaddition, it may be time consuming for a user/client to have to downloadan HTML file, pass it to a pre-processing program, and then load it intoa web browser.

A different embodiment utilizing a proxy server can eliminate some ofthe difficulties of the above-described embodiment. A proxy server is aserver computer that receives Hypertext Transport Protocol (HTTP, theunderlying communication protocol of the World-Wide Web) requests from aclient computer and performs that request on its behalf. Proxy serversare often used to allow Web access beyond a firewall; the proxy serverhas special privilege to access the Internet beyond the firewall, so allHTTP requests must go through it. Proxy support is common feature in webbrowsers and is extremely easy for a user to configure.

FIG. 16 is a block diagram illustrates the use of a proxy server forforce feedback functionality over the Web, in which generic effects areemulated with authored effects. To solve the pre-processing problem, aninterested party, company, organization, etc. can set up a proxy server400 connected to the Internet 402. This proxy server 400 accepts allHTTP requests from client computers, such as requests to receiveparticular web pages. For example, a host computer 404 is to receive aweb page from a desired web server 406. The host 404 issues a request408 to the proxy server 400. The proxy server 400 then issues a request410 on the host's behalf, which is received by the requested source 406,such as a web server computer over the Web. The web server 406 thenfetches the desired document/web page and sends the document 412 to theproxy server 400.

After receiving the requested document 412, a normal proxy server wouldthen pass the requested web page back to the client; however, the proxyserver 400 of the present invention first applies the pre-processing(described above) to the web page document to modify the web page file(embedding a force-only ActiveX control, adding JavaScript code tofacilitate force-effect scripting, and assigning effects to web pageobjects such as hyperlinks), if the document is an HTML file. The proxyserver may receive and read user preferences from the client todetermine which force effects to assign to which web page objects. Ifthe document is not an HTML file, no preprocessing is performed. Theproxy server 400 then transmits back the new generic-effect-enabled webpage 414 (or other type of document if no preprocessing was performed)to the client computer 404. Thus, the client's user is able to feel theweb page in a seamless manner, and without having the client expend timein preprocessing operations.

Plug-In Embodiment

Another embodiment for implementing force effects is to provide a“plug-in” extension to the web browser at the client machine, such asNetscape Navigator. The plug-in, for example, can implement authoredforce effects, where the plug-in handles particular force-relatedcommands or information in a received HTML file. A plug-in may also oralternatively implement generic force effects for a web page byreferring to the generic effects on the client machine when anappropriate event takes place, such as a pointer moving over ahyperlink. The plug-in software can be a proprietary extension of theweb browser software developed, for example, by Immersion Corporation. Aplug-in is similar in function to the ActiveX control described above.

In one embodiment, instructions are provided in the received web pagewhich define an authored force effect for the plug-in. For example, an<EMBED . . . > tag can define a force button object that will bedisplayed on the client machine. Other force objects besides buttonobjects can also be defined and displayed, such as links, text, sliders,game objects (balls, paddles, etc.), avatars, windows, icons, menu bars,drop-down menus, or other objects. In a first line of the <EMBED . . . >command, the force button object can be defined by a “IFF” extensionfile, namely “FORCEBUTTON.IFF.” The <EMBED . . . > command is anexisting functionality of HTML. It essentially embeds function callswhich are handled by the web browser. If the suffix of the specifiedfile is a known, standard suffix type, the call is executed directly bythe web browser. If, however, the suffix (.IFF in this instance) is nota standard feature of the web browser, the browser will first look for a“plug-in” to implement this feature and, if a suitable plug-in is notfound, it will look for application programs implementing this feature.In the preferred embodiment of the present invention, a plug-inincluding a reference to a Dynamically Linked Library (DLL) is providedto give functionality to the IFF suffix. The DLL can be provided localto the client machine or on another linked resource.

In the <EMBED . . . > specification, the size of the button can bespecified, the initial state of the button (“up” or “down”), and theforce effect associated with the button, such as “vibration.” Parametersdefining the character and nature of the force effect can also bespecified (start time, length, frequency, magnitude, etc.). A “trigger”for the force effect can be specified, such as a function “MOUSEWITHIN”with its associated parameters, and by a function “BUTTONSTATE.” Thefunction MOUSEWITHIN determines whether the pointer icon, the positionof which is controlled by the force feedback device, is within thespecified boundaries defining a region of the force button. This regioncan be specified by the parameters. The function BUTTONSTATE detennineswhether a button or switch of the force feedback device is in thedesired state to trigger the force object event (e.g., a button event inthis example). The image file and text representing the force button canalso be specified. Other force effects, triggers and parameters can alsobe associated with the force object. For example, a force (such as avibration) can be triggered if the pointing icon is moved apredetermined velocity or within a predefined range of velocities withinthe force object. Or, a trajectory of the pointing icon on a forceobject can trigger a force, like a circle gesture.

The plug-in embodiment also provides for programmability of the embeddedforce feedback object. An optional programmable command can be insertedinto the EMBED command and can include, for example, an iterative loopto cause a force effect to be repeated a specified number of timesand/or to cause a wait for a specified time after a force effect hasoccurred. By providing programmability to the force feedback object,force feedback effects based upon past events and upon a complexinteraction of factors can be provided.

If there is an embedded “tag” for a force effect in the HTML file, e.g.a tag or file having an IFF reference, then the plug-in software is usedto interpret the IFF file. In one embodiment, a “framework” is createdfor the force effect (or “force object”). The framework provides aparticular set of generic features to implement the specified forceobject, and preferably includes no specific parameters or functions forthe force object. A name and associated parameters from the HTML fileare then parsed and the force object is built upon this framework basedupon the parameters. For example, one name might be “BUTTONSTATE” andits parameter might be “UP” (or “UNSELECTED”).

The plug-in software can monitor the position and button state of theforce feedback device. The plug-in creates a force feedback command inresponse to the detected position and state (if appropriate), and acommand is sent to the Dynamically Linked Library (DLL) to place a forcefeedback command on the interface which can be parsed and interpreted bythe force feedback device.

It should be noted that the force feedback driver (browser plug-in orDLL) can have the ability to interact with Java code. In thisembodiment, the plug-in reads and executes Java commands using thebrowser's run-time Java interpreter. It should also be noted that theforce feedback device itself can have a Java interpreting chip on board,permitting the plug-in driver to download Java code to the forcefeedback device to be executed on the device. Java and Java interpretingchips are available under license from SUN Microcomputers of MountainView, Calif.

The embodiments disclosed herein can have the ability to interact withinstructions provided in other languages besides HTML. For example,virtual reality 3-D graphical environments are increasingly beingcreated and implemented over the World Wide Web and Internet usinglanguages such as the Virtual Reality Modeling Language (VRML). In these3-D graphical environments, users may interact with programmed 3-Dobjects and constructs using client computer 14 or 16, and may alsointeract with 3-D graphical representations (or “avatars”) controlled byother users over the World Wide Web/Internet from other clientcomputers. Force feedback commands and parameters can be provided in theinstructions or files of these other protocols and languages andreceived by a client computer system in an equivalent manner to thatdescribed above so that force feedback can be experienced in simulated3-D space. For example, embedded force feedback routines for authoredforce effects can be included in the VRML data for a virtual environmentso that when the user moves into a virtual wall, an obstruction force isgenerated on the user-manipulatable object. Or, when the user carries avirtual object in a controlled virtual glove, the user might feel asimulated weight of the virtual object on the user manipulatable object.In such an embodiment, the force feedback device preferably provides theuser with three or more degrees of freedom of movement so that input inthree dimensions can be provided to the client computer. Alternatively,generic effects can be provided with VRML files having standard objecttypes (walls, ramps, stairs, ice, avatars, gloves, chairs, guns, etc.)

Authoring Tool for Adding Force Functionality to Web Pages

Associated with the methods of providing web pages having force effectsas described herein are methods for allowing authors to easily createweb pages with force effects. A web page author does not have anintuitive sense as to how forces will feel when assigned to particularobjects or adjusted in certain ways, and thus must go to great effort todevelop characteristics of forces that are desired for a specific objector interaction. For example, a programmer may wish to create a specificspring and damping force sensation between two graphical objects on aweb page, where the force sensation has a particular stiffness, play,offset, etc. Current web page authoring tools do not allow the design offorce effects associated with web page objects. Furthermore, currentforce effect design tools are not oriented to the particularities of webpage authoring, such that a web page designer is without a professionaldevelopment tool to assist in authoring force effects.

The web authoring application interface described herein is one exampleof a force-enabled tool for authoring web pages. For example, theinterface herein (“FEELit Studio”) is described in connection with theI-Force Studio® available from Immersion Corporation, which is aforce-effect authoring interface intended for use with applicationprograms such as game programs. This interface can be used with mouse 36(e.g., the FEELit Mouse developed by Immersion Corporation). Theapplication allows users inexperienced with HTML syntax and unfamiliarwith force feedback to quickly create HTML pages with feel content,along with standard content such as graphics, text, and sound. Theapplication uses common user interface features found in most webauthoring applications today. These features should allow the user toeasily insert and place content and immediately feel, see, and hear theresults. Once the web page is saved as a file by the authoringinterface, the user can send the created web page to a web server to beaccessible to end users over the Internet or other network; or, thesaved web page might be already available over the Internet if it isresident on a machine (such as a client machine) having appropriateaccessibility over the Internet (or other network).

Adding force effects to web pages can be very simple. However, to addeffective force effects, the feel sensations should appropriately matchthe rest of the web page content. This requires a creative andinteractive process where parameters are defined, experienced, andmodified until the feel sensations are fine tuned. A clumsy andnon-intuitive design process means the web author will spend much lesstime being creative.

The web page authoring tool of the present invention is used forauthoring web pages that include force effects. In one embodiment, theweb authoring tool is an HTML editor, such as an editor that is providedwith a web browser, with additional force functionality.

FIG. 17a is a screenshot of an HTML editor 429 that has been modifiedaccording to the present invention to include the ability to design andadd forces to web pages according to the present invention. The HTMLeditor is a WYSIWYG (What You See Is What You Get) editor for creating aweb page, meaning the author sees exactly how the web page will look toothers as the author edits the page. The author may type text directlyinto the editor as it will appear, and can place images and other webobjects using GUI tools. The author may assign text, an image, or otherobject as a hyperlink using menu controls. The HTML editor then outputsthe finalized web page as an HTML file that includes all the requiredHTML instructions necessary to implement the created web page layout.This GUI method of creating a web page is in contrast to creating a webpage directly using HTML code. When using the editor, the obtuse syntaxof the HTML document is hidden from the author as well as the end user.

For example, the logo and image on the web page shown in FIG. 17 isprovided with HTML code specifying the position of the logo and imageand references the image file used to display the image. The WYSIWYGHTML editor provides a more intuitive method of creating the logo andpicture. The user simply places the cursor where he or she desires thepicture to appear on the web page and then selects the “Insert Image”command from either the “Insert” pull-down menu or the right-clickcontext menu. A dialog box is displayed and allows the author to specifyan image file. A similar process is used to make the image a hyperlinkto another web page. An HTML editor which visually shows what the pagewill look like rather than display the underlying source text used toconstruct the page is much easier to use and understand. The same can besaid for adding feel sensations.

The HTML editor of the present invention not only is a WYSIWYG editorfor visual design of web pages; it is also a WYSIWYF (What You See IsWhat You Feel) editor, meaning the author immediately sees and feelsexactly how the web page will feel to others as the author edits thepage. Just as the obtuse syntax of the HTML document is hidden from theauthor, the mechanism enabling feel content in the web page should beabstracted from the author. This mechanism can be ActiveX and JavaScriptcomponents or some other technology such as the methods described above.The author is presented with a set of controls and tools that allowsquick and simple addition of compelling feel content without having tounderstand the underlying technology. Authors may easily assign feelproperties to all web content, including graphics and text. Designedforce effects can be saved as a resource and can be reloaded andmodified. End-users can also customize the sensations they feel on webpages authored with the authoring tool.

For example, as shown in FIG. 17a, a “Properties” dialog box 432 can bedisplayed in the HTML editor when the image 430 of the mouse oh the webpage is selected and the properties menu from a drop-down menu isselected. This dialog box 432 is used to add to or change properties ofthe selected object. Thus, an image file can be associated with theselected object (in this case, “mouse.gif”) using the “image” tab 434,the object can be designated a link object and the linked location isspecified in the input field using the “link” tab 436, or the placing ofthe image or associated text can be changed using the “paragraph” tab438.

A fourth tab 440 of the present invention is named “Feel” and isprovided in addition to the “Image,” “Link,” and “Paragraph” tabs. TheFeel tab can be selected to bring up a new screen, page, or dialog boxpresenting a number of options for the author to select in designingforces to add to the image 430. This feel property page would allow theuser to customize the feel of the image using one or more force effects.A list of force effects can be presented from which the user can selectone or more. Also, in some embodiments a full design interface as shownin FIGS. 19a-19 c can be presented to allow a user to design forces aswell as assign them to web page objects.

FIG. 17b illustrates another example of providing options for the userto assign a force effect for a web page object. In web page authoringtool 444, the user has selected the web page object 445, an image, andhas commanded the force and sound assignment window 447 to open, e.g.with a click of a right mouse button or the selection of a menu item.Assignment window 447 allows the user to assign one or more forceeffects from a list 448. In the example shown, the user has selected“BumpyRoad” force effect 451.

The user may also select via the control menu 452 the conditions or“events” required for one or more selected force effects to be “played”,i.e., output using the force feedback device 50. For example, thecontrol menu 452 allows the user to select whether the force effect willbe output when the mouse cursor is over the associated web page object,or when the user clicks a button of the mouse while the cursor is overthe object, or when then the user double-clicks a button of the mousewhile the cursor is over the object. Other events related to the webpage object can also be assigned and/or defined by the user. A togglebutton 443 allows the user to select whether a force continues to “play”after the cursor moves out of the boundaries of the web page object. Forexample, some time-based force effects have a particular duration afterbeing started that can continue after the cursor leaves the object.

The user can also select a different “resource” file using button 453 toprovide a different list 448 of force effects; each resource file, forexample, can include different force effects. A resource file or forceeffect in list 448 can be stored on a local client machine (e.g. forgeneric effects or downloaded authored effects) or can be storedremotely over a local-area network or the Internet and downloaded asnecessary. The content of a force effect file is typically much morecompact in size than the content of image or sound files, since a forceeffect need only be characterized in a force effect file usingparameters such as magnitude and duration. The parameters are utilizedby resources already resident on the client machine to output thecharacterized force. For location-based force effects requiring moreinput than simply selecting a force effect, another assignment windowcan be displayed. For example, if an enclosure force effect is desiredto be assigned to web object 445, the user could draw the exact shapeand size of the enclosure in an interface as shown below in FIGS. 20-24.Or, the user can designate the exact location on the web page displayedin the interface 444 where an attractive force is to be centered.

A test button 442 preferably allows the user to test a selected forceeffect from the list 448. For example, the Bumpyroad force effect 451would cause the mouse grasped by the user to immediately shake after thetest button was selected. For location-based forces such as an enclosureor attractive force, the user can test the force effect by moving thecursor to the selected web page object 445 as will be displayed in theweb page, as if the web page had been downloaded (in some embodiments,all force effects having a “cursor over object” event can also be testedthis way). For example, if an enclosure surrounded image 445, the usercould test the enclosure forces by moving the cursor into and out of thedisplayed image 445. The bumpy road effect 451 can be tested in someembodiments by moving the cursor over the object if the event 452requires such.

The user can also preferably associate audio feedback with web pageobjects using the web page authoring tool of the present invention,where the audio effects are synchronized with any force effects of theweb page object. For example, in editor 444, a list of sound files 446is displayed which are available to be associated with the web pageobject 445. These sound files can be stored on a client machine or on adifferent machine accessible over a network, and are preferably in astandard format, such as wav or mp3. The add button 448 allows theselected sound file to be associated with the selected web page object,and the sound will be played when the event in window 452 occurs (anyassigned force effect 451 will also begins at the designated event). Thetest button 450 allows the selected sound file to be played immediatelyin the authoring tool (assuming output speakers are connected to thecomputer displaying the web page authoring application). Once a soundfile has been associated with a web page object, the output of the soundis automatically synchronized with the output of the associated forceeffect. Since many force effects have much less impact or immersivenesson a user without sound, this feature allows straightforward associationand synchronization of force effects and audio effects.

An example of HTML code for a web page including an image such as image445, which has been associated with an “engine” force effect and an“engine” sound, follows. This code uses JavaScript to call a force-onlyActiveX control, as described above. The force-enabled authoring toolpreferably automatically generates this code when the user wishes tosave the created web page.

<!DOCTYPE HTML PUBLIC “−//W3C//DTD W3 HTML//EN”> <HTML> <HEAD> <METAcontent=“text/html; charset=iso-8859-1”http-equiv=Content-Type> <METAcontent=‘“MSHTML 4.72.3110.7”’ name=GENERATOR> </HEAD> <BODYonload=“openforceresource (‘F:/web/Sounds.ifr’);”> <object ID=“_engine”WIDTH=“0” HEIGHT=“0”CLASSID=“CLSID:05589FA1-C356-11CE-BF01-00AA0055595A”> <paramname=“ShowControls” value=“0”> <param name=“ShowDisplay” value=“0”><param name=“AutoStart” value=“0”> <param name=“AutoRewind” value=“1”><param name=“FileName” value=“F:\web\sound\engine.wav”> </object><P>.</P> <OBJECT classid=CLSID:5DFDD466-5B37-11D1-A868-0060083A2742codeBase=FeelMe.CAB#version=2,0,0,0 height=1 id=FeelObject width=1><PARAM NAME=“_Version” VALUE=“65536”> <PARAM NAME=“_ExtentX” VALUE=“26”><PARAM NAME=“ExtentY” VALUE=“26”> <PARAM NAME=“StockProps” VALUE=“0”></OBJECT> <SCRIPT id=“FEELit Base JavaScript” language=JavaScript> varloaded = false; function openforceresource(url) {FeelObject.OpenForceResource(url); loaded = true; } functiondoeffect(force, file) { if (loaded) { var result =FeelObject.StartEffect(force, file); } { function noeffect(force, file){ if (loaded) { var- result = FeelObject.StopEffect(force, file); } }function forcecleanup( ) { if (loaded) { FeelObject.Cleanup( ); loaded =false; } } </SCRIPT> <P>&nbsp;</P> <P><IMG align=baseline alt=“ ”border=0 hspace=0 onclick=“ ” ondblclick=“ ” onmouseout=“ ”onmouseover=“doeffect(‘Engine’); _engine.Stop( ); _engine.Run( ); ”src=“F:\web\feelit.gif”></P> <P align=right id=FeelMeLogo> <Ahref=“http://www.immerse.com/”> <IMG alt=“FEEL ME!” height=31src=“FeelMeMini.gif” width=50></A></P> </BODY> </HTML>

The web page authoring tool 444 also preferably allows an author to adda graphical identifier and/or link to the created web page thatindicates that the web page includes force feedback functionality. Forexample, the identifier can be a particular logo (e.g., text or agraphical image) that is easily recognized. This identifier can beautomatically added at a particular predefined/customized location onthe web page, such as the corner or the bottom center of the page, etc.,when the web page is written to a file using the authoring tool of thepresent invention, if the web page includes one or more force effects(or the identifier can be included in any web page written by theforce-enabled authoring application). The identifier can also be of apredefined, customized size. When a user of a client machine downloadsthe force feedback web page, the force identifier is thus also displayedso that the user knows whether force-feedback peripherals are supportedby the web page; if desired, the identifier might also indicate thatauthored force effects are included in the web page. Furthermore, insome embodiments the graphical identifier can be a hyperlink that, whenselected with the cursor, will cause a force-feedback related website tobe downloaded on the client machine. The force feedback website canprovide force feedback resources such as additional force effects,guides, updates, etc. The user can preferably set an option as towhether the identifier will automatically be included in a web pagecreated by the web editing tool.

The force-enabled web page authoring tool of the present invention isinteractive, intuitive, and allows users to instantly feel exactly whatthey create. For example, after a user assigns a force effect to animage in the web page, the user can test the force effect by simplymoving the cursor over that image while the web page is displayed in theauthoring tool If the force effect is not quite what the user wanted, adifferent force effect can be immediately assigned. In some embodiments,the user can enter a “test” mode of the authoring tool to feel forceeffects, which is separate from an editing mode.

FIG. 18 is a block diagram illustrating the web page authoring interfaceof the present invention and an example of a web page structure producedby the interface. The authoring interface 444 (or 429) creates or editsone or more web pages 454, where multiple web pages can be loaded intothe interface at one time if desired. Each web page 454 includes one ormore web page objects 455, such as images, text, animations, etc. Eachweb page object can include one or more events 456, which are theconditions needed to play or activate an effect assigned to the event,such as a cursor located over the object or a click of a mouse button.Thus, each event 456 can be assigned to one or more effects, such asforce effects 457 or sound effects 458 (also, a single force effect orsound effect can be assigned to multiple different events). As shown,events can be assigned force effects and no sound effects, sound effectsand no force effects, or both force effects and sound effects. Inaddition, each web page 454 can preferably be assigned one or moreglobal events 459. Global events are not associated with a particularweb page object 455, but are applied universally. Each global event hasone or more force effects 457 and/or one or more sound effects 458associated with it, similar to the events 456. For example, one globalevent can occur when the web page is fully downloaded by a client, atwhich point the assigned force effect(s) and sound(s) are played. Adifferent global event can occur when the web page is exited by theclient machine. Other global events can be a mouse click or double-clickwhen the cursor is not over any web page object (e.g., over thebackground), when a key on the keyboard is pressed, etc.

FIGS. 19a, 19 b, and 19 c illustrate examples of force sensation designinterfaces that can be displayed to edit the force effects for an objectin the HTML editor. Alternatively, a separate force design application,such as I-Force Studio, can be run if the user wishes to modify orcreate a force effect. The shown interface windows are similar to theforce sensation design tools provided in I-FORCE Studio® available fromImmersion Corporation. These design tools provide a fully animatedgraphical environment for rapidly adjusting physical parameters of feelsensations, and then optionally saving them as “feel resources.” Authorsmay craft tactile feel sensations by stretching springs, manipulatingsurfaces, placing liquids, and adjusting other graphical representationsand physical metaphors that represent each associated force feedbackphenomenon. The design tools also empower end users with the ability toedit the “feel resource” using the same intuitive animated graphicalcontrols used by the web page author. A user with no programmingexperience or familiarity with force feedback can quickly designhigh-fidelity, compelling sensations using these design tools. Suchgraphical manipulation for design of force effects is described ingreater detail in U.S. Pat. Nos. 6,147,674 and 6,169,540, bothincorporated herein by reference.

FIG. 19a shows a number of windows for adjusting parameters for severaldifferent force effects. The author can select a desired force effectfrom the displayed list 460 of effects, and may select one or morebuttons on the force feedback device 50 from list 461 to be associatedwith the force effect. Window 463 shows a list of force effectsorganized under the “Compound” force effect main heading in list 460,any of which can be selected by the web page author (or user). Examplesof windows for force effects that have been selected from list 460 and463 are also shown. Window 462 shows the parameters and graphicalrepresentation of an “earthquake” compound force effect. Window 464shows the parameters and graphical representation of an “explosion”compound force effect. Window 466 shows the parameters and graphicalrepresentation of a “slope” force effect (i.e. spring force with anegative coefficient). In all of these windows, the author may adjustthe parameters by entering values into fields or by adjusting variousportions of the graphical representation, e.g., dragging control pointson the displayed “earthquake” wave 468 to adjust amplitude andfrequency. Parameters like frequency, duration, attack and fade levels,and direction are also adjustable by dragging graphical controls. Theauthor may adjust the slope effect by balancing the ball 469 on theslope. In all these cases (assuming the author has a force feedbackdevice 50 connected to the computer on which the force effects aredesigned), the author may immediately and directly feel the designedforce effect on the mouse 50 by pressing an appropriate “output”control.

FIG. 19b shows other force effects that can be designed or adjusted bythe user. A feeling of water or mud can be easily designed in the damperwindow 470 by adjusting the liquid level and then feeling viscosityforces on the mouse 36 change as the simulated liquid 471 flows out ofthe graphical container. Window 472 shows the design of an “angle wall”that provides a barrier force to the user object at a specified angle,and which can be adjusted using a graphical representation of the wall.When adjustments are made to the parameters, the author immediatelyfeels the new effect on mouse 36 and can judge if the changes enhancethe feel or not. Apply and Undo commands allow the user to easily keepor discard changes. A user can save, copy, modify, or combine predefinedeffects to create a library of favorite feel sensations. The authoringtool also provides fully customizable sensations and numerous presetexamples: custom sensations can be created or predefined effects can beused or modified.

FIG. 19c shows examples of a design window 474 for designing anenclosure force effect and design window 476 for an attract/repel forceeffect. The various force characteristics of the walls of the enclosurecan be designed, and the interior of the enclosure can be assigned aforce effect using button 475. The attract/repel force effect can becharacterized by magnitude, conditions for activation or deactivation,radius/range, etc. Any designed forces can be immediately tested bymoving the cursor within the design window over or near the appropriategraphical representation 477. The forces shown in FIGS. 19a and 19 b arenot generally referenced to a screen location; for example, the outputof a vibration or a damping force does not depend on the location of apointer on a screen. However, the force effects of FIG. 19c and manyother web-related and graphical-user-interface related forces areassociated with a particular screen location. For example, an enclosureis a box- or elliptical-shaped object having walls at a particularscreen location which provide forces on a manipulandum based on thelocation of a cursor with respect to the enclosure walls. An attractionforce typically attracts the cursor and manipulandum to a particularlocation on the screen. The user should be able to immediately testthese types of screen-location forces using a cursor in the designinterface. Thus, in one embodiment, the location with reference to theframe of a particular force design window should be converted to fullscreen coordinates so that the location of the graphical representation477 of the force effect is known to the force feedback device and forcescan be output at the appropriate location when the cursor interacts withthe graphical representation 477.

The design interface includes intuitive graphical metaphors so that theuser may adjust force feedback parameters using easy-to-understandcontrols. It is fully interactive and animated, so that the force effectparameters can be adjusted by dragging the pictorial control points,adjusting sliders and dials, or typing in numbers directly. Theresulting sensations are output in real-time to the interface device 50as the user manipulates the parameters. If a user designs a slipperyforce effect, he/she can immediately run the cursor over arepresentation to feel the slipperiness. If the feeling is too strong ortoo subtle, there is a simple control or dialog box to adjust theslipperiness and immediately feel the changed effect. The design tooluses real world metaphors like the liquid, slope, and walls in the aboveinterface to ensure users unfamiliar with force feedback will have anintuitive understanding of the functions of all the controls.

In addition, a force design interface such as shown in FIGS. 19a-19 c(e.g. I-Force Studio from Immersion Corp.) preferably is provided withthe ability to allow users to test audio feedback that is to beassociated with one or more force effects. Preferably, a user candesignate audio data, such as a “wav” file, mp3 file, or other sounddata, to be associated with a designed force effect. This can beaccomplished by presenting a list of audio files and selecting one ormore to be associated with a particular audio effect. The user can then“play” the force effect to test it, and the audio effect will play insynchronization with the force effect, i.e. the sound will start whenthe force effect starts (for a force effect having a duration); or thesound is played whenever the force is output (for some continuous forceeffects). Thus a user may iteratively adjust a force to match thevariances in a sound as the sound is played, allowing the user to adjustthe force effect and/or the sound for more effective presentation andmutual interaction.

FIGS. 20-25 illustrate a web authoring feature of the present inventionto conveniently add force sensations to web pages in a completelycustomizable, spatially-related fashion. FIG. 20 shows a portion of aweb browser editor 480 that is displaying a web page 482 being edited bya web page author. The web page includes text and an image 484. Theauthor wishes to quickly add force sensations to the image in particularareas of the image.

FIG. 21 illustrates the same web page and the use of a drawing tool toadd force sensations to an image by defining spatial features of theadded feel. The author controls cursor 486 to draw an outline 488 aroundthe image of the truck portion of the image 484. The outline can bedrawn using any number of well-known functionality tools of drawing orCAD programs (i.e. line drawing tool, freehand tool, an algorithm thatautomatically outlines a desired object in a bitmap, etc.). In addition,the author has drawn outlines 490 around the wheel portions of the truckimage, and has drawn interior lines 492 to represent other forcesensations. These outlines are intended to define the areas on the webpage where forces are to be felt by a user with a mouse pointer. Theoutlines and lines are invisible to a user who has downloaded the webpage and displayed it normally in a web browser, but are discernedhaptically by the user with a force feedback device 50.

The outlines and lines of image 484 drawn by the author are shown moreclearly in FIG. 22 without the image. Border 494 represents the extentof the entire image 484. Body outline 488 and wheel outlines 490 may beassociated with force sensations by the author using the web pageeditor; for example, a particular outline can be selected and one ormore force effects can be assigned to the outline or to a particularinteraction with the outline. For example, outline 488 and outlines 490can be designated by the author to be enclosures having barrier forceson the borders of the enclosures that provide a slight wall resistanceto the mouse 36. When a user downloads the web page to a client machineover the Internet, the user can move his or her mouse pointer to theborders of outlines 488 and 490 and feel the outline of the truck andits wheels. In another example, the enclosures may be associated withgravity forces when the pointer is moving into the enclosure andvibration forces when the pointer is moving out of the enclosure. Theauthor can assign a different force effect to each portion or side of anoutline, the interior of the outline, a portion of an outline, etc.Force effects may be defined that occur externally to the outline butare caused by a pointer or button interaction with the outline.

The lines 492 drawn in the interior of the outline 488 are correlatedwith borders of windows on the truck image and may be associated, forexample, with “bumps” or barrier forces (i.e. the mouse 36 is subjectedto a bump force when the pointer 486 is moved over a line 492). Thisallows a user who interacts with the web page in a browser to feelphysical features of the truck while he or she moves the pointer insidethe outline 488. The user can likewise draw an outline within outline488 to provide an enclosure (which is preferably a closed shape). Forexample, the windshield of the truck can be provided with a smooth orice texture (which causes forces to bias or amplify movement of themouse 36 within the enclosure and cause difficulty in stopping the mousemovement as if slipping on ice) while the other interior areas ofenclosure 488 are provided with a bumpier texture.

By simply drawing lines or contours in desired places over the image484, the author can add forces at a desired location or region on theweb page. This allows careful correlation between spatial location offeel sensations and images or portions of images. In addition, theauthor can quickly designate multiple areas inside images, text, orother objects which may have different force effects associated withthem. This feature allows greater complexity of forces to be assigned toweb pages to a level of detail greater than the level of web pageobjects. Preferably, an author is also able to draw similar force linesor regions on any portion of the web page, e.g. on the background, ontext, on link objects, on animations/video objects, etc.

FIGS. 23 and 24 illustrate another example of the use of the spatialforce editor described above. In FIG. 23, web browser editor 500 isdisplaying a web page 502 which includes an image 504 that an authorwishes to edit to add force sensations. In FIG. 24, the author has usedpointer 486 to draw an outline 506 around the jeans portion 508 of image504. The author then assigns the outline 506 a force effect such as atexture. For example, a corduroy or rough-cloth type texture can beprovided so that a user who view the web page will feel forces thatsimulate a cloth-like texture when the mouse 36 is moved to move thepointer over the jeans portion 508. However, when the user's pointer ismoved over the other (non-jeans) portions of image 504 (such as thewhite background areas 510), no texture will be felt. The outliningfeature allows a web page author to designate particular areas withinimages or other objects to have desired forces.

In another embodiment, the force feedback web page editor of the presentinvention can include an “auto trace” function which will automaticallyoutline a desired sub-region in an image or other object. For example,some existing software such as Streamline® from Adobe SystemsIncorporated will automatically trace areas or objects within bitmapswhich the user designates to create an outline or boundary so that theuser need not manually trace the areas. In the present invention, theauthor can cause an area in a bitmap to be outlined automatically andthen assign force effects to the outline.

It should be noted that users may modify downloaded web pages in asimilar fashion as described above to add force sensations to areas onweb pages. The outlines 488, 490, and 506 and lines 492 are visible to auser or author when the web page is displayed by an HTML (or other)editor having the required force functionality, and the images andforces may be manipulated by anyone using the editor. The outlines andlines are preferably invisible when the web page is displayed in normaldownloading/browsing mode by a web browser (unless, for example, theviewer desires to see the outlines and enables a “view force outlines”option in a web browser).

FIG. 25 summarizes an embodiment in which a single all-in-one webauthoring application (the HTML editor shown) contains fully functionalforce design tools which allow the user to design feel sensationsdirectly within the application in addition to designing the text andgraphics of the page.

A web authoring application 520 includes an HTML editor which can importimages, add text, animations, or other web objects as allowed byconventional web page authoring tools. Application 520 also includesforce assigning capabilities according to the present invention, so thatimages, text, or other web page objects can be assigned forceproperties. For example, an image object can be selected and thenassigned a force effect, as shown in FIGS. 17a and 17 b. In addition,sound files can also be associated with the web page objects, as shownin FIG. 17b.

Furthermore, force design tools are also included in the application520. The forces can be designed in child force design tool “palettes”522 which are part of the parent application 520. The palettes eachinclude a feel editor with graphical controls in a design window, whereeach palette is for designing a different type of force effect, like thedifferent force effect windows shown in FIGS. 19a-19 c. The user candesign textures, vibrations, and other force effects by adjustinggraphical and numeric parameters; such parameters can include effecttype, magnitude, direction, type-specific parameters (e.g. frequency,spacing), duration, triggers by button-press or mouse-over, and theactive region of an object (the default active region coincides withtext box/image borders, but can also be customized with user-specifiedcoordinates and borders, such as borders being outside, inside, ornonexistent and having specified direction and thickness). To implementand allow the user to test those types of force effects in which thelocation on the screen of the graphical representation needs to be known(e.g. for enclosures, attraction/repulsion forces, etc. such as shown inFIG. 19c), the design tool palettes 522 work directly in theapplication's client coordinates (i.e. the window of the palette) with aconversion function to provide full screen coordinates that aretypically used directly with an API and force feedback mouse (similar tothe coordinate conversion described with reference to FIG. 8). Interfacebuttons allow the user to experiment with different force effects whendesigning the effects for the web page object. Thus, the user caniteratively design force effects quickly and easily by testing theeffects directly in the design interface.

The application 520 outputs a force-enabled HTML file 523 which can bestored on a server and downloaded by a client machine. The force effectcontent (parameters, etc.) that were designed or modified in the forcedesign palettes can be saved directly in the HTML file along with allthe other text, graphics, and other non-force content, and nointermediate force effect files are required. In other embodiments, aforce effect file can be created using the design palettes 522 and theproduced HTML file can included reference the separate force effectfiles which are downloaded with or after the web page. For example, theHTML file can include Javascript which references a force-only ActiveXcontrol, a media ActiveX control to enable any sounds, force effectfiles (unless force content is included in the HTML), and sound files.Other formats and implementations can also be used for the HTML file. Inaddition, there can be an option provided in the authoring tool to allowstandard HTML code to be generated without force information.

FIG. 26 illustrates an alternative embodiment of a force feedback webpage authoring tool, in which a stand-alone force design application 524is provided which only is capable of designing feel sensations, and astand-alone force-enabled web authoring tool 526 for designing the webpage including graphics, text, and links is separately used. The forcedesign application 524 includes a feel editor with graphical controls,such as shown above in FIGS. 18 and 19, where the user can design forceeffects and adjust parameters to test the effects. An example of such aneditor is I-Force Studio from Immersion Corp. The parameters can includeeffect type, magnitude, direction, type-specific parameters (e.g.frequency, spacing), and duration. Screen-position force effects such asenclosures are referenced in client (window) coordinates, which arereadily portable to other applications after converting them to fullscreen coordinates. The application 524 can save out force effects asforce effect files 528, which can be standardized with a file extension(e.g. “IFR”), for example.

The separate web authoring application 526 references one or more forceeffect files 528 that were created by the force design application 524.The web authoring application provides a force-enabled HTML editor inwhich web page objects may be assigned force properties. The application526 allows a web page object to be assigned a force effect stored in theforce effect file 528 that was designed in the design application 524,and to associate sound files with the force effects. For example, theeditor of FIGS. 17a or 17 b can be utilized as the standaloneapplication 526 that imports a resource file including one or more forceeffects as shown in FIG. 17b. If a user wishes to design or edit a forcewhile the authoring tool 526 is running, the force design applicationcan immediately be run and displayed as well. The authoring tool of thepresent invention thus includes seamless web authoring integration withother force feedback design tools such as I-Force Studio.

Application 526 is used to save a web page 527 in HTML code, where theweb page 527 includes tags or references to force effect files and soundfiles, much the same way that standard HTML includes references to imagefiles. Alternatively, the HTML file can include the force content asdescribed above. In one preferred embodiment, the application 526 savesan HTML file including Javascript code, references to used ActiveXcontrols, a name of a force effect file and a name of a force effectwithin the file, and a name of a sound file (if applicable). TheJavascript code preferably accesses a force-only ActiveX control toenable forces (as described above) and, for example, accesses adifferent media ActiveX control (such as ActiveMovieControl fromMicrosoft) to enable sounds when a user downloads the force-enabled webpage. Events can be provided as event handlers in the HTML code thatpass information to the JavaScript instructions. Other implementationscan also be used, for example as described herein.

The web authoring tool of FIG. 25 or 26 can use differentimplementations to add force effects to HTML code. For example, in oneembodiment, the authoring tool takes an existing HTML document (orcreates a new document), internally uses Dynamic HTML to obtain objectinformation and add force effects. As explained. above, the authoringtool can output a web page in normal HTML including additional code forimplementing the force effects (such as JavaScript which calls aforce-only ActiveX control).

When a web browser of a client machine is used to download theforce-enabled web page produced by authoring tool of FIG. 25 or FIG. 26,the browser parses the HTML and JavaScript to display the web page andimplement the forces and sounds. The ActiveX controls referenced by theJavaScript are typically already resident on the client machine or maybe downloaded as needed. The force effect files and sound filesreferenced in the HTML are downloaded (similarly to downloading anyreferenced images in standard HTML) or may already be resident on theclient.

End User Customization of Feel Content

Another type of force design capability can be provided to allow an enduser, rather than just an author of a web page, to customize forceeffects for all web pages or for a particular web page. The genericforce effects described herein allow end users to experience genericfeel sensations for web page objects of a particular type. For example,all hyperlinks can feel like gravity wells, and all images can feel likea glass surface. Ideally, the user on a client machine should be able toadjust the set of generic feel effects to suit his or her preferences.For example, the user may want all hyperlinks to generate a vibrationinstead of a gravity well. Customization ability already exists in mostbrowsers for characteristics like the color of hyperlinks—the webbrowser automatically assigns the hyperlink a color based on the userpreference. However, web authors can override those default colors tomake the hyperlinks match a web page's color scheme. Two possible waysof allowing end users to customize the set of generic feel effects aredescribed below.

One method is to allow the user to adjust user preferences with acentral control panel, either associated with their local GUI forcepreferences or with the Web browser preferences. This control panelcould allow a force file to be specified as the desired force effect fora specific type of web page object. The force file can include all theparameters and data necessary to characterize the desired force effect.In addition, a force file can specify force effects for a whole set ofdifferent web page objects, or all the objects on a particular web page.Furthermore, in some embodiments as described above, different sets ofgeneric force effects can be specified for different web pages. Thus, aset of generic force effects could apply only to a particular associatedweb page.

Another method for feel customization is to encapsulate all the forcedesign functionality in an ActiveX object (or other programmingconstruct) embedded in the HTML page, as described in greater detailabove. In this case, any user browsing a web page with the Force DesignActiveX object embedded within has the force design tools readilyavailable to customize all the feel content on the web page. The userwould not need to have a force design web authoring interface or aforce-enabled web authoring application installed to perform thiscustomization. For example, a web page that includes a texture and avibration references a force-only ActiveX object that enables andoutputs these two forces on the client machine. A user interface canalso be included in the ActiveX object downloaded to the client machineto allow a user to edit the texture force or the vibration force. Theuser interface need not be fully functional, since it is downloaded withthe web page (and bandwidth may need to be conserved); and the userinterface need not enable the user to edit/design all possible or evenmost types of forces, just the forces provided on the web page withwhich it is included. Alternatively, a more fully functional user designinterface can be provided in the downloaded ActiveX object.

The authoring tools described herein have focussed on allowing webcontent authors to easily add compelling feel content to their HTMLpages. These HTML pages would then be seen, heard, and felt by end usersover the WWW. However, end users may also be interested in using thesetools, even though they may not be authoring web pages. For end users,these tools can also offer a means of customizing the feel content inexisting web pages to suit their preferences. Such a feature is relevantin light of the fact that some browsers, such as Microsoft's InternetExplorer 4, provides an “Active Desktop”, which treats the entireoperating system on the client machine as a web page. Therefore, the webpage authoring tools can conceivably be used to customize the feel offiles, folders, icons, scroll bars, buttons, windows, and other objectsdisplayed in a GUI of an operating system.

While this invention has been described in terms of several preferredembodiments, there are alterations, permutations, and equivalents whichfall within the scope of this invention. It should also be noted thatthere are may alternative ways of implementing both the process andapparatus of the present invention. For example, although specificreference is made to the World Wide Web, the features of the presentinvention can be provided with use of other networks and networkprotocols. In addition, the features described herein can be combinedpartially or wholly with one another if desired to achieve a desiredparticular implementation.

What is claimed is:
 1. A method for providing force effects for a webpage, the method comprising: determining which of a plurality of webpage objects are to be associated with at least one force effect, saidweb page objects to be displayed in said web page, wherein said web pageis derived from web page information received from a server machine overa network, said web page information including screen displayinformation representing said web page objects; selecting a force effectto associate with a particular web page object based on a type of saidparticular web page object and based on a mapping that associates saidtype of web page object with said force effect, said mapping beingstored on a client machine that received said web page; and sending aforce signal to a force feedback interface device when a user-controlledcursor interacts with said particular web page object, said cursor andsaid force web page objects being displayed on said web page by saidclient machine, wherein said force signal is output to an actuator of aforce feedback interface device coupled to said client machine to causea force sensation to a user of said force feedback interface device, andwherein said force signal is based on said force effect.
 2. A method asrecited in claim 1 wherein a force control program running on saidclient machine detects whether said user-controlled cursor is contactingone of said particular web page objects having said type.
 3. A method asrecited in claim 2 wherein said force control program is a separateapplication from a web browser displaying said web page on said clientmachine.
 4. A method as recited in claim 1 wherein said mapping isstored as preference information on said client machine.
 5. A method asrecited in claim 1 wherein said web page information is processed to addforce information to said web page to implement said force effect, saidforce information being added before said web page is displayed by saidclient machine.
 6. A method as recited in claim 5 wherein said clientmachine receives said web page information from said server, determineswhich of said web page objects are associated with force effects, andprocesses said web page information to add force information to said webpage to implement said force effect.
 7. A method for providing forceeffects for a web page, the method comprising: enabling a determinationof whether a web page object is associated with one or more authoredforce effects, wherein effect information specifying said authored forceeffects is included in web page information received by a client machinefrom a server machine over a network, wherein said web page informationalso includes screen display information representing said web pageobject to be displayed in a web page on said client machine; enabling adetermination of whether a web page object is associated with one ormore generic force effects, wherein effect information specifying saidgeneric force effects is stored on said client machine prior toreceiving said web page information, and wherein said generic forceeffects are each applicable to a predefined type of web page object; andenabling a providing of force information when a user-controlled cursorinteracts with a particular web page object on said web page associatedwith at least one of said authored force effects or associated with atleast one of said generic force effects, wherein said force informationcauses a force signal to be output to a force feedback interface devicecoupled to said client machine and causes an actuator of said forcefeedback interface device to output a force sensation to a user of saidforce feedback interface device, and wherein said force information isbased on said effect information associated with said particular webpage object.
 8. A method as recited in claim 7 wherein if said web pageincludes both an authored force effect and a generic force effect,further comprising enabling an override of said authored force effectwith said generic force effect such that said generic force effect isoutput instead of said authored force effect.
 9. A method as recited inclaim 8 wherein said authored force effect is overridden based onpreferences of said user for said generic force effects, saidpreferences stored on said client machine.
 10. A method as recited inclaim 7 wherein if said web page includes both an authored force effectand a generic force effect, further comprising overriding said genericforce effect with one of said authored force effects if said genericforce effect and said authored force effect are associated with the sameweb page object.
 11. A method as recited in claim 7 wherein said effectinformation for said authored force effect includes informationspecifying that said generic effect information specifying said genericeffects and stored on said client machine is to be used in implementingsaid authored force effect.
 12. A method as recited in claim 7 whereinsaid authored force effect is implemented using an ActiveX controlreferenced by said effect information for said authored force effect.13. A method as recited in claim 7 wherein properties of said web pageobject associated with said generic force effect are accessed usingDynamic HTML.
 14. A method for implementing a web page authoringinterface including the ability to add force sensations to a web page,the method comprising: enabling a display of said web page authoringinterface on a display device of a host computer, said web pageauthoring interface displaying a web page including a plurality of webpage objects; enabling a reception of input from a user to said web pageauthoring interface, said input selecting one of said web page objectsand selecting a force effect to be associated with said selected webpage object; and enabling an output of a web page including said webpage objects and including force information to allow said force effectto be implemented when said web page is displayed by a client machineafter being received by said client machine from a server machine over anetwork, wherein said force effect is commanded by said client machineand output as a force sensation by a force feedback interface devicecoupled to said client machine, said force feedback interface deviceincluding a user manipulatable object graspable and moveable by a userof said client machine.
 15. A method as recited in claim 14 wherein saidforce information is a reference to a separate force effect file,wherein said force effect file is downloaded by said client machine tooutput said force effect using said force feedback interface device. 16.A method as recited in claim 14 wherein said force information includesforce effect content data characterizing said force effect.
 17. A methodas recited in claim 14 wherein said web page authoring interfaceincludes a force design interface for creating or modifying said forceeffect.
 18. A method as recited in claim 15 further comprising creatingsaid force effect in said force effect file using a force designinterface separate from said web page authoring interface.
 19. A methodas recited in claim 14 wherein said force effect is output to a forcefeedback device and experienced by said user of said web page authoringinterface when a user-controlled displayed cursor interacts with saidweb page object assigned to said force effect.
 20. A method as recitedin claim 14 wherein said web page is in HTML format.
 21. A method asrecited in claim 20 wherein said web page includes scriptinginstructions for accessing a force-only control program.
 22. A method asrecited in claim 14 further comprising receiving spatial input from saiduser to said web page authoring interface, said spatial input spatiallydesignating an area of one of said web page objects to be associatedwith a selected force effect.
 23. A method as recited in claim 14wherein said user of said web page authoring interface may select acondition in said design interface, said condition determining when saidforce effect is output based on user input during display of said webpage.
 24. A method as recited in claim 23 wherein said conditionincludes a cursor controlled by said user moving over said web pageobject on said web page.
 25. A method as recited in claim 23 whereinsaid condition includes a button on said user manipulatable object beingpressed by said user.
 26. A method as recited in claim 14 furthercomprising receiving input from said user of said web page authoringinterface to associate sound data with said selected web page object,such that when said force effect is output, a sound defined by saidsound data is output to said user synchronized with said force effect.27. A method as recited in claim 26 wherein when said force effect isoutput, a start of said output sound is synchronized to a start of saidforce effect output.
 28. A method for providing force effects for a webpage, the method comprising: enabling a reception of web pageinformation from a server machine over a network, said web pageinformation including screen display information representing aplurality of web page objects to be displayed in said web page; enablinga determination of which of said web page objects are to be associatedwith at least one force effect, wherein said web page objects associatedwith said at least one force effect have a predefined type; and enablingan association of a generic force effect with each of said web pageobjects having said predefined type, each of said generic force effectsbeing defined by associated effect information derived from a clientmachine, wherein a generic force effect associated with a particular oneof said web page objects causes a force signal to be output by saidclient machine when a user-controlled cursor interacts with saidparticular web page object, said cursor and said web page objects beingdisplayed on said web page by said client machine, wherein said forcesignal causes an actuator of a force feedback interface device coupledto said client machine to output a force sensation to a user of saidforce feedback interface device, and wherein said force signal is basedon said effect information associated with said generic force effectthat is assigned to said particular web page object.